Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 37,939 of 193,264
Downloads: 33,784

Daily

Ranking: 12,573 of 192,861
Downloads: 68

Depended by

RankDownloadsName
16,166123,191hostelify
29,48947,684site_analyzer
96,4108,842rwspider

Depends on

RankDownloadsName
81,312,927,961rake
5,3781,304,766echoe

Owners

#GravatarHandle
1iconrinzi