Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,328 of 189,409

Downloads: 349,215

Daily

Ranking: 18,271 of 189,386

Downloads: 11

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,169	39,215	cohesion
61,288	17,153	applied_css
161,689	3,170	seo_site_check

Depends on

Rank	Downloads	Name
9	1,231,849,099	rake
12	1,194,926,783	json
15	1,137,817,917	nokogiri
18	1,113,323,107	addressable
20	1,105,744,275	rspec-core
29	948,688,129	rspec
80	532,306,113	redis
136	330,247,824	sinatra
157	299,979,104	sidekiq
229	203,307,132	thin
239	196,535,790	redis-namespace
245	189,777,911	haml
311	151,680,271	slop
995	47,509,300	mock_redis
1,876	16,052,576	coveralls
2,233	10,156,493	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee