Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,845 of 196,338

Downloads: 360,304

Daily

Ranking: 22,527 of 196,320

Downloads: 3

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,891	40,823	cohesion
62,487	17,742	applied_css
114,616	2,297	rawl
164,555	3,312	seo_site_check

Depends on

Rank	Downloads	Name
8	1,399,913,164	rake
9	1,360,651,456	json
15	1,252,875,222	nokogiri
16	1,252,388,300	addressable
20	1,190,642,606	rspec-core
29	1,013,199,560	rspec
83	578,647,847	redis
146	354,447,713	sinatra
157	333,004,662	sidekiq
254	210,719,158	redis-namespace
259	208,311,049	thin
265	204,596,714	haml
336	157,408,329	slop
950	54,432,027	mock_redis
1,915	17,219,613	coveralls
2,145	12,878,006	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee