Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,240 of 188,252

Downloads: 347,064

Daily

Ranking: 9,401 of 188,232

Downloads: 118

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,957	38,970	cohesion
60,961	17,047	applied_css
161,290	3,138	seo_site_check

Depends on

Rank	Downloads	Name
10	1,188,890,086	rake
12	1,160,896,542	json
16	1,108,614,704	nokogiri
18	1,088,470,577	rspec-core
21	1,080,119,455	addressable
29	935,425,343	rspec
79	520,572,225	redis
134	324,409,136	sinatra
157	293,148,912	sidekiq
223	202,086,548	thin
236	193,162,467	redis-namespace
243	186,488,529	haml
304	150,613,456	slop
995	46,364,822	mock_redis
1,842	15,931,730	coveralls
2,237	9,767,329	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee