Categories: None [Edit]
cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Total
Ranking: 10,017 of 198,774
Downloads: 362,337
Daily
Ranking: 23,881 of 198,739
Downloads: 11
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
| Rank | Downloads | Name |
|---|---|---|
| 34,365 | 41,058 | cohesion |
| 63,267 | 17,858 | applied_css |
| 114,616 | 2,297 | rawl |
| 166,030 | 3,351 | seo_site_check |
Depends on
| Rank | Downloads | Name |
|---|---|---|
| 8 | 1,469,998,038 | rake |
| 9 | 1,441,042,198 | json |
| 15 | 1,314,289,146 | addressable |
| 17 | 1,304,159,190 | nokogiri |
| 20 | 1,231,703,059 | rspec-core |
| 29 | 1,043,852,410 | rspec |
| 83 | 600,312,341 | redis |
| 150 | 364,901,972 | sinatra |
| 158 | 349,114,492 | sidekiq |
| 264 | 216,515,785 | redis-namespace |
| 277 | 210,979,890 | haml |
| 279 | 210,280,683 | thin |
| 347 | 160,198,504 | slop |
| 938 | 57,628,719 | mock_redis |
| 1,931 | 17,609,475 | coveralls |
| 2,096 | 14,341,404 | bundle-audit |
Owners
| # | Gravatar | Handle |
|---|---|---|
| 1 | stewartmckee |