Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,186 of 187,571

Downloads: 344,612

Daily

Ranking: 9,552 of 187,548

Downloads: 113

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,832	38,654	cohesion
60,743	16,909	applied_css
161,012	3,089	seo_site_check

Depends on

Rank	Downloads	Name
10	1,165,196,519	rake
13	1,135,784,591	json
16	1,088,415,844	nokogiri
18	1,074,662,374	rspec-core
22	1,056,546,369	addressable
28	924,848,930	rspec
79	513,282,788	redis
133	320,279,969	sinatra
157	287,826,461	sidekiq
220	201,377,757	thin
233	190,762,964	redis-namespace
240	184,197,072	haml
303	149,733,456	slop
998	45,484,236	mock_redis
1,818	15,845,822	coveralls
2,233	9,508,386	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee