Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 8,570 of 183,052

Downloads: 332,208

Daily

Ranking: 17,121 of 183,043

Downloads: 7

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
31,792	36,564	cohesion
59,039	16,085	applied_css
158,862	2,834	seo_site_check

Depends on

Rank	Downloads	Name
10	951,441,545	rake
14	938,249,346	rspec-core
18	908,175,174	json
20	874,585,621	nokogiri
23	859,590,309	addressable
25	816,694,487	rspec
72	442,915,794	redis
122	281,354,665	sinatra
144	244,976,660	sidekiq
209	169,971,170	thin
215	164,303,726	redis-namespace
226	157,623,483	haml
255	140,813,777	slop
978	38,169,156	mock_redis
1,703	14,081,533	coveralls
2,230	7,508,527	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee