Categories: None [Edit]

robotstxt

https://rubygems.org/gems/robotstxt
https://github.com/rinzi/robotstxt
Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Total

Ranking: 37,700 of 188,339
Downloads: 32,525

Daily

Ranking: 37,464 of 188,315
Downloads: 5

Depended by

RankDownloadsName
15,646121,372hostelify
28,98546,448site_analyzer
95,3218,650rwspider

Depends on

RankDownloadsName
101,191,537,815rake
5,2931,237,607echoe

Owners

#GravatarHandle
1iconrinzi