Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,525 of 192,414

Downloads: 353,395

Daily

Ranking: 8,862 of 192,395

Downloads: 138

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,658	39,635	cohesion
61,812	17,381	applied_css
162,538	3,230	seo_site_check

Depends on

Rank	Downloads	Name
8	1,303,222,835	rake
12	1,259,869,844	json
15	1,184,242,827	nokogiri
16	1,171,304,292	addressable
21	1,139,584,789	rspec-core
29	974,273,805	rspec
82	551,844,759	redis
139	339,802,850	sinatra
157	313,107,985	sidekiq
240	205,357,399	thin
244	202,437,061	redis-namespace
254	195,725,082	haml
321	153,990,063	slop
980	50,131,249	mock_redis
1,900	16,454,249	coveralls
2,191	11,208,026	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee