Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,287 of 188,757

Downloads: 348,095

Daily

Ranking: 15,830 of 188,742

Downloads: 23

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,067	39,125	cohesion
61,146	17,110	applied_css
161,566	3,154	seo_site_check

Depends on

Rank	Downloads	Name
10	1,202,260,445	rake
12	1,175,331,415	json
15	1,121,927,753	nokogiri
19	1,095,998,648	rspec-core
20	1,094,814,215	addressable
29	941,173,382	rspec
80	525,444,141	redis
135	327,127,343	sinatra
157	296,072,165	sidekiq
227	202,660,937	thin
237	194,559,879	redis-namespace
244	187,986,949	haml
307	151,121,762	slop
996	46,842,522	mock_redis
1,861	15,986,457	coveralls
2,231	9,941,648	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee