Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 37,718 of 188,757
Downloads: 32,721

Daily

Ranking: 21,788 of 188,742
Downloads: 9

Depended by

RankDownloadsName
15,690121,694hostelify
29,04246,625site_analyzer
95,4278,688rwspider

Depends on

RankDownloadsName
101,202,260,445rake
5,2831,256,377echoe

Owners

#GravatarHandle
1iconrinzi