Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 37,809 of 193,262
Downloads: 34,162

Daily

Ranking: 15,473 of 193,241
Downloads: 28

Depended by

RankDownloadsName
16,215123,508hostelify
29,50047,946site_analyzer
96,5608,871rwspider

Depends on

RankDownloadsName
81,331,347,704rake
5,3961,310,215echoe

Owners

#GravatarHandle
1iconrinzi