Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 34,084 of 198,774
Downloads: 41,524

Daily

Ranking: 16,967 of 198,739
Downloads: 34

Depended by

RankDownloadsName
16,793125,079hostelify
30,45048,529site_analyzer
98,7609,007rwspider

Depends on

RankDownloadsName
81,469,998,038rake
5,6031,340,091echoe

Owners

#GravatarHandle
1iconrinzi