Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,975 of 198,025

Downloads: 361,707

Daily

Ranking: 35,772 of 197,989

Downloads: 3

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
34,242	40,999	cohesion
63,082	17,826	applied_css
114,616	2,297	rawl
165,781	3,334	seo_site_check

Depends on

Rank	Downloads	Name
8	1,452,458,801	rake
9	1,419,663,551	json
15	1,297,636,047	addressable
17	1,289,467,455	nokogiri
20	1,220,486,059	rspec-core
29	1,035,635,780	rspec
83	594,427,585	redis
149	362,179,488	sinatra
158	344,692,034	sidekiq
263	214,947,149	redis-namespace
273	209,786,422	thin
275	209,258,145	haml
342	159,387,949	slop
939	56,853,294	mock_redis
1,923	17,536,096	coveralls
2,100	14,020,110	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee