Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,303 of 188,892

Downloads: 348,269

Daily

Ranking: 14,983 of 188,868

Downloads: 26

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,092	39,132	cohesion
61,179	17,115	applied_css
161,576	3,158	seo_site_check

Depends on

Rank	Downloads	Name
10	1,208,584,337	rake
12	1,179,693,528	json
15	1,125,720,410	nokogiri
19	1,099,155,032	addressable
20	1,098,285,846	rspec-core
29	942,932,199	rspec
80	527,056,867	redis
135	327,871,849	sinatra
157	297,007,678	sidekiq
227	202,816,637	thin
237	195,018,595	redis-namespace
245	188,408,503	haml
308	151,257,137	slop
994	47,000,864	mock_redis
1,865	16,000,830	coveralls
2,230	9,993,781	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee