Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,261 of 188,368

Downloads: 347,311

Daily

Ranking: 17,033 of 188,354

Downloads: 31

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,982	39,006	cohesion
60,990	17,068	applied_css
161,324	3,145	seo_site_check

Depends on

Rank	Downloads	Name
10	1,192,998,841	rake
12	1,165,280,852	json
16	1,112,285,569	nokogiri
18	1,090,801,521	rspec-core
21	1,084,383,831	addressable
29	937,210,484	rspec
79	521,804,637	redis
134	325,177,743	sinatra
157	294,069,437	sidekiq
224	202,226,575	thin
236	193,562,933	redis-namespace
244	186,891,806	haml
306	150,752,187	slop
995	46,518,545	mock_redis
1,846	15,946,833	coveralls
2,236	9,819,757	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee