Categories: None [Edit]
cobweb
Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
Total
Ranking: 8,741 of 184,211
Downloads: 335,506
Daily
Ranking: 28,870 of 184,203
Downloads: 2
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
Rank | Downloads | Name |
---|---|---|
32,093 | 36,960 | cohesion |
59,459 | 16,254 | applied_css |
159,417 | 2,884 | seo_site_check |
Depends on
Rank | Downloads | Name |
---|---|---|
10 | 999,352,732 | rake |
14 | 968,811,263 | rspec-core |
17 | 961,488,489 | json |
20 | 924,702,300 | nokogiri |
22 | 904,705,521 | addressable |
26 | 840,445,034 | rspec |
72 | 461,252,191 | redis |
124 | 291,901,564 | sinatra |
152 | 255,613,286 | sidekiq |
209 | 179,168,589 | thin |
218 | 171,960,901 | redis-namespace |
228 | 164,694,198 | haml |
270 | 142,772,145 | slop |
995 | 40,041,828 | mock_redis |
1,767 | 14,191,453 | coveralls |
2,225 | 8,013,156 | bundle-audit |
Owners
# | Gravatar | Handle |
---|---|---|
1 | stewartmckee |