Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,431 of 191,066

Downloads: 351,261

Daily

Ranking: 11,144 of 191,018

Downloads: 24

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,320	39,398	cohesion
61,511	17,234	applied_css
161,939	3,198	seo_site_check

Depends on

Rank	Downloads	Name
8	1,265,791,545	rake
12	1,227,116,763	json
15	1,161,867,800	nokogiri
17	1,142,633,559	addressable
20	1,122,260,141	rspec-core
29	961,592,452	rspec
80	542,850,509	redis
138	334,887,909	sinatra
157	306,392,942	sidekiq
233	204,318,239	thin
241	199,622,678	redis-namespace
252	192,718,245	haml
316	152,747,766	slop
992	48,822,948	mock_redis
1,892	16,199,399	coveralls
2,216	10,690,390	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee