Categories: None [Edit]
cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Total
Ranking: 8,100 of 180,402
Downloads: 323,508
Daily
Ranking: 7,464 of 180,392
Downloads: 188
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
| Rank | Downloads | Name |
|---|---|---|
| 31,155 | 35,457 | cohesion |
| 58,117 | 15,627 | applied_css |
| 157,579 | 2,730 | seo_site_check |
Depends on
| Rank | Downloads | Name |
|---|---|---|
| 12 | 875,086,170 | rspec-core |
| 16 | 851,212,111 | rake |
| 17 | 808,527,169 | json |
| 21 | 786,927,932 | nokogiri |
| 23 | 766,708,551 | rspec |
| 24 | 763,369,799 | addressable |
| 70 | 404,377,579 | redis |
| 117 | 259,485,526 | sinatra |
| 142 | 221,486,883 | sidekiq |
| 206 | 150,449,906 | thin |
| 209 | 148,619,436 | redis-namespace |
| 219 | 142,870,704 | haml |
| 230 | 136,620,900 | slop |
| 959 | 34,081,719 | mock_redis |
| 1,617 | 13,285,513 | coveralls |
| 2,255 | 6,391,312 | bundle-audit |
Owners
| # | Gravatar | Handle |
|---|---|---|
| 1 |  | stewartmckee |
xmisao