Categories: None [Edit]

cobweb

https://rubygems.org/gems/cobweb
https://github.com/stewartmckee/cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 10,017 of 198,774
Downloads: 362,337

Daily

Ranking: 23,881 of 198,739
Downloads: 11

Depended by

RankDownloadsName
34,36541,058cohesion
63,26717,858applied_css
114,6162,297rawl
166,0303,351seo_site_check

Depends on

RankDownloadsName
81,469,998,038rake
91,441,042,198json
151,314,289,146addressable
171,304,159,190nokogiri
201,231,703,059rspec-core
291,043,852,410rspec
83600,312,341redis
150364,901,972sinatra
158349,114,492sidekiq
264216,515,785redis-namespace
277210,979,890haml
279210,280,683thin
347160,198,504slop
93857,628,719mock_redis
1,93117,609,475coveralls
2,09614,341,404bundle-audit

Owners

#GravatarHandle
1iconstewartmckee