Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,917 of 197,085

Downloads: 361,260

Daily

Ranking: 8,203 of 197,066

Downloads: 98

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
34,076	40,932	cohesion
62,809	17,796	applied_css
114,616	2,297	rawl
165,418	3,323	seo_site_check

Depends on

Rank	Downloads	Name
8	1,428,256,062	rake
9	1,392,216,932	json
15	1,276,740,491	addressable
16	1,272,378,399	nokogiri
20	1,206,687,339	rspec-core
29	1,025,379,194	rspec
83	586,869,473	redis
147	358,601,712	sinatra
158	339,062,734	sidekiq
257	213,031,096	redis-namespace
266	209,142,504	thin
270	207,158,634	haml
342	158,450,240	slop
943	55,744,100	mock_redis
1,917	17,418,617	coveralls
2,133	13,387,208	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee