Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,084 of 186,698

Downloads: 340,528

Daily

Ranking: 31,699 of 186,685

Downloads: 11

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,785	37,593	cohesion
60,373	16,517	applied_css
160,403	2,981	seo_site_check

Depends on

Rank	Downloads	Name
10	1,125,994,016	rake
13	1,094,772,179	json
17	1,052,381,120	rspec-core
18	1,051,305,334	nokogiri
22	1,018,059,901	addressable
27	908,151,619	rspec
79	501,543,504	redis
131	313,313,011	sinatra
153	279,722,692	sidekiq
216	197,661,537	thin
229	187,162,329	redis-namespace
236	180,388,435	haml
293	148,273,152	slop
996	44,158,065	mock_redis
1,795	15,698,828	coveralls
2,224	9,103,321	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee