Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 37,653 of 187,967
Downloads: 32,357

Daily

Ranking: 23,770 of 187,960
Downloads: 19

Depended by

RankDownloadsName
15,603121,058hostelify
28,92546,242site_analyzer
95,1248,616rwspider

Depends on

RankDownloadsName
101,178,214,454rake
5,3101,214,964echoe

Owners

#GravatarHandle
1iconrinzi