Categories: None [Edit]

robotstxt-parser

https://rubygems.org/gems/robotstxt-parser
https://github.com/gjtorikian/robotstxt-parser
Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 13,034 of 186,947
Downloads: 165,481

Daily

Ranking: 69,506 of 186,938
Downloads: 8

Depended by

RankDownloadsName

Depends on

RankDownloadsName
101,139,345,512rake
2,9804,989,031fakeweb

Owners

#GravatarHandle
1icongjtorikian