Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 8,617 of 183,521

Downloads: 333,253

Daily

Ranking: 37,317 of 183,406

Downloads: 2

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
31,884	36,703	cohesion
59,161	16,143	applied_css
159,047	2,847	seo_site_check

Depends on

Rank	Downloads	Name
10	966,727,118	rake
14	947,649,003	rspec-core
18	924,997,167	json
20	887,455,148	nokogiri
23	873,626,564	addressable
25	823,958,774	rspec
72	448,670,911	redis
124	284,634,231	sinatra
145	248,362,033	sidekiq
209	172,821,074	thin
215	166,695,261	redis-namespace
226	159,799,945	haml
262	141,423,545	slop
983	38,806,351	mock_redis
1,724	14,117,514	coveralls
2,232	7,671,080	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee