Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,156 of 187,239

Downloads: 343,613

Daily

Ranking: 14,683 of 187,224

Downloads: 35

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,788	38,503	cohesion
60,605	16,849	applied_css
160,781	3,069	seo_site_check

Depends on

Rank	Downloads	Name
10	1,151,901,046	rake
13	1,122,458,692	json
15	1,077,086,433	nokogiri
18	1,067,325,014	rspec-core
22	1,044,220,967	addressable
27	919,404,019	rspec
79	509,510,097	redis
132	318,128,948	sinatra
156	285,170,445	sidekiq
217	200,987,871	thin
232	189,612,451	redis-namespace
238	182,989,042	haml
299	149,277,550	slop
996	45,057,791	mock_redis
1,810	15,786,188	coveralls
2,227	9,371,685	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee