Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,888 of 196,658

Downloads: 360,908

Daily

Ranking: 34,119 of 196,619

Downloads: 9

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,984	40,876	cohesion
62,578	17,777	applied_css
114,616	2,297	rawl
165,036	3,317	seo_site_check

Depends on

Rank	Downloads	Name
8	1,416,600,134	rake
9	1,379,284,867	json
15	1,266,623,349	addressable
16	1,264,176,847	nokogiri
20	1,200,171,163	rspec-core
29	1,020,526,382	rspec
83	583,408,591	redis
147	356,897,451	sinatra
158	336,561,216	sidekiq
255	212,096,869	redis-namespace
260	208,792,662	thin
267	206,084,502	haml
338	158,025,007	slop
945	55,160,394	mock_redis
1,915	17,360,233	coveralls
2,136	13,156,270	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee