Categories: None [Edit]
cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Total
Ranking: 8,096 of 180,404
Downloads: 323,132
Daily
Ranking: 102,511 of 180,385
Downloads: 1
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
| Rank | Downloads | Name |
|---|---|---|
| 31,158 | 35,424 | cohesion |
| 58,143 | 15,615 | applied_css |
| 157,610 | 2,728 | seo_site_check |
Depends on
| Rank | Downloads | Name |
|---|---|---|
| 12 | 873,779,464 | rspec-core |
| 16 | 849,094,940 | rake |
| 17 | 806,503,044 | json |
| 21 | 785,016,323 | nokogiri |
| 23 | 765,651,376 | rspec |
| 24 | 761,334,426 | addressable |
| 70 | 403,573,383 | redis |
| 117 | 259,020,900 | sinatra |
| 142 | 221,007,426 | sidekiq |
| 206 | 149,960,421 | thin |
| 208 | 148,322,718 | redis-namespace |
| 219 | 142,594,119 | haml |
| 230 | 136,526,736 | slop |
| 959 | 34,003,264 | mock_redis |
| 1,616 | 13,269,309 | coveralls |
| 2,254 | 6,369,071 | bundle-audit |
Owners
| # | Gravatar | Handle |
|---|---|---|
| 1 |  | stewartmckee |
xmisao