Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,932 of 197,258

Downloads: 361,370

Daily

Ranking: 41,708 of 197,235

Downloads: 2

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
34,115	40,948	cohesion
62,908	17,803	applied_css
114,616	2,297	rawl
165,514	3,325	seo_site_check

Depends on

Rank	Downloads	Name
8	1,434,061,178	rake
9	1,398,402,316	json
15	1,281,599,163	addressable
16	1,276,322,770	nokogiri
20	1,209,785,317	rspec-core
29	1,027,618,561	rspec
83	588,594,759	redis
147	359,382,358	sinatra
158	340,343,695	sidekiq
257	213,478,665	redis-namespace
269	209,291,355	thin
274	207,667,316	haml
342	158,659,241	slop
942	56,029,485	mock_redis
1,920	17,438,704	coveralls
2,132	13,487,435	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee