Categories: None [Edit]
cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Total
Ranking: 9,802 of 195,762
Downloads: 359,540
Daily
Ranking: 15,754 of 195,747
Downloads: 37
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
| Rank | Downloads | Name |
|---|---|---|
| 33,810 | 40,730 | cohesion |
| 62,389 | 17,699 | applied_css |
| 114,616 | 2,297 | rawl |
| 164,099 | 3,304 | seo_site_check |
Depends on
| Rank | Downloads | Name |
|---|---|---|
| 8 | 1,384,293,602 | rake |
| 9 | 1,343,682,665 | json |
| 15 | 1,241,927,384 | nokogiri |
| 16 | 1,238,797,314 | addressable |
| 20 | 1,181,672,217 | rspec-core |
| 29 | 1,006,320,288 | rspec |
| 83 | 574,047,675 | redis |
| 145 | 352,114,429 | sinatra |
| 157 | 329,587,740 | sidekiq |
| 252 | 209,331,576 | redis-namespace |
| 254 | 207,836,674 | thin |
| 264 | 203,164,513 | haml |
| 334 | 156,838,678 | slop |
| 959 | 53,686,364 | mock_redis |
| 1,912 | 17,099,844 | coveralls |
| 2,152 | 12,590,258 | bundle-audit |
Owners
| # | Gravatar | Handle |
|---|---|---|
| 1 |  | stewartmckee |
xmisao