Categories: None [Edit]

cobweb

https://rubygems.org/gems/cobweb
https://github.com/stewartmckee/cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 8,741 of 184,211
Downloads: 335,506

Daily

Ranking: 28,870 of 184,203
Downloads: 2

Depended by

RankDownloadsName
32,09336,960cohesion
59,45916,254applied_css
159,4172,884seo_site_check

Depends on

RankDownloadsName
10999,352,732rake
14968,811,263rspec-core
17961,488,489json
20924,702,300nokogiri
22904,705,521addressable
26840,445,034rspec
72461,252,191redis
124291,901,564sinatra
152255,613,286sidekiq
209179,168,589thin
218171,960,901redis-namespace
228164,694,198haml
270142,772,145slop
99540,041,828mock_redis
1,76714,191,453coveralls
2,2258,013,156bundle-audit

Owners

#GravatarHandle
1iconstewartmckee