Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,206 of 187,950

Downloads: 345,863

Daily

Ranking: 8,600 of 187,922

Downloads: 216

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
32,908	38,794	cohesion
60,831	16,983	applied_css
161,138	3,113	seo_site_check

Depends on

Rank	Downloads	Name
10	1,176,878,843	rake
12	1,148,066,961	json
16	1,098,107,829	nokogiri
18	1,081,370,221	rspec-core
21	1,067,509,828	addressable
28	929,993,045	rspec
79	516,900,927	redis
134	322,324,818	sinatra
157	290,456,730	sidekiq
221	201,728,911	thin
235	191,964,330	redis-namespace
241	185,333,681	haml
303	150,168,353	slop
997	45,906,543	mock_redis
1,837	15,884,575	coveralls
2,234	9,629,154	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee