Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,113 of 186,947

Downloads: 342,203

Daily

Ranking: 7,429 of 186,938

Downloads: 141

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,773	38,065	cohesion
60,468	16,699	applied_css
160,575	3,025	seo_site_check

Depends on

Rank	Downloads	Name
10	1,139,345,512	rake
13	1,108,513,326	json
16	1,064,716,963	nokogiri
18	1,060,120,092	rspec-core
22	1,031,233,253	addressable
27	913,996,669	rspec
79	505,565,527	redis
131	315,947,519	sinatra
154	282,425,129	sidekiq
217	199,310,047	thin
230	188,409,837	redis-namespace
238	181,714,935	haml
294	148,795,691	slop
996	44,616,595	mock_redis
1,805	15,744,733	coveralls
2,223	9,233,012	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee