Categories: None [Edit]

cobweb

https://rubygems.org/gems/cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 5,191 of 153,772
Downloads: 191,941

Daily

Ranking: 7,309 of 153,744
Downloads: 94

Depended by

RankDownloadsName
23,30420,144cohesion
43,9329,247applied_css
111,6312,146rawl
135,9641,171seo_site_check

Depends on

RankDownloadsName
1459,148,232rspec-core
5433,948,717rspec
11289,761,984rake
12270,643,943json
18240,183,383nokogiri
35165,394,889addressable
67101,123,464redis
7793,082,359sinatra
8482,677,091slop
12457,135,096thin
14049,514,201haml
15144,592,285redis-namespace
15344,340,503sidekiq
8716,993,628coveralls
1,0594,621,399mock_redis
3,949335,694bundle-audit

Owners

#GravatarHandle
1iconstewartmckee