Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,680 of 192,968

Downloads: 354,661

Daily

Ranking: 21,509 of 192,934

Downloads: 16

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,695	39,776	cohesion
61,917	17,439	applied_css
162,736	3,249	seo_site_check

Depends on

Rank	Downloads	Name
8	1,321,873,897	rake
12	1,275,718,133	json
15	1,195,434,009	nokogiri
16	1,184,855,607	addressable
20	1,147,779,415	rspec-core
29	980,384,371	rspec
82	556,249,706	redis
140	342,209,644	sinatra
158	316,432,612	sidekiq
244	205,849,331	thin
247	203,802,673	redis-namespace
257	197,191,392	haml
324	154,650,391	slop
977	50,830,546	mock_redis
1,905	16,588,749	coveralls
2,186	11,500,325	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee