Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,385 of 189,920

Downloads: 350,194

Daily

Ranking: 6,108 of 189,899

Downloads: 118

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,250	39,278	cohesion
61,396	17,185	applied_css
161,785	3,182	seo_site_check

Depends on

Rank	Downloads	Name
8	1,247,280,359	rake
12	1,209,347,443	json
15	1,149,155,520	nokogiri
18	1,125,953,396	addressable
20	1,113,047,914	rspec-core
29	954,351,217	rspec
80	537,119,714	redis
136	332,309,411	sinatra
157	302,898,992	sidekiq
230	203,781,529	thin
240	197,946,447	redis-namespace
248	191,051,188	haml
312	152,131,671	slop
996	48,061,449	mock_redis
1,885	16,110,254	coveralls
2,226	10,363,913	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee