Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 38,122 of 192,156
Downloads: 33,359

Daily

Ranking: 25,668 of 192,137
Downloads: 17

Depended by

RankDownloadsName
16,024122,902hostelify
29,38747,574site_analyzer
96,1868,812rwspider

Depends on

RankDownloadsName
81,293,584,300rake
5,3561,294,673echoe

Owners

#GravatarHandle
1iconrinzi