Categories: None [Edit]

cobweb

https://rubygems.org/gems/cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Total

Ranking: 4,992 of 151,987
Downloads: 189,098

Daily

Ranking: 6,323 of 151,947
Downloads: 107

Depended by

RankDownloadsName
22,84819,766cohesion
43,2099,096applied_css
110,8122,069rawl
134,5571,116seo_site_check

Depends on

RankDownloadsName
1429,916,224rspec-core
5406,929,584rspec
11270,570,798rake
12251,426,783json
18222,034,617nokogiri
37147,286,632addressable
6890,280,812redis
7385,256,436sinatra
8177,778,157slop
11854,080,409thin
13846,609,532haml
14840,601,769redis-namespace
15539,031,634sidekiq
8106,697,070coveralls
1,0374,232,418mock_redis
4,327254,107bundle-audit

Owners

#GravatarHandle
1iconstewartmckee