Categories: None [Edit]

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 9,467 of 191,816

Downloads: 352,243

Daily

Ranking: 14,289 of 191,796

Downloads: 37

Downloads Trends

Ranking Trends

Num of Versions Trends

Popular Versions (Major)

Popular Versions (Major.Minor)

Depended by

Rank	Downloads	Name
33,445	39,481	cohesion
61,698	17,317	applied_css
162,345	3,210	seo_site_check

Depends on

Rank	Downloads	Name
8	1,284,528,619	rake
12	1,244,236,784	json
15	1,173,788,716	nokogiri
17	1,157,524,271	addressable
20	1,131,597,533	rspec-core
29	968,366,586	rspec
81	547,618,127	redis
139	337,453,693	sinatra
157	309,898,143	sidekiq
237	204,866,968	thin
242	201,123,181	redis-namespace
253	194,306,538	haml
319	153,407,011	slop
986	49,475,280	mock_redis
1,892	16,373,610	coveralls
2,204	10,942,004	bundle-audit

Owners

#	Gravatar	Handle
1		stewartmckee