Categories: None [Edit]
scrapetor
Scrapetor is a Ruby HTML parsing + scraping toolkit. The parser is a native C arena DOM with structural indexes built at parse time and NEON SIMD scanners in the SAX hot loop. A streaming extraction engine compiles the schema DSL into a single forward pass — no DOM materialised, one Ruby boundary crossing per document. On builds where libcurl is available, Scrapetor::Fetcher adds an HTTP/2-capable fetch layer with per-thread connection cache, shared DNS + TLS session pool, in-process gzip / deflate / brotli / zstd decoding, iconv charset transcoding, retry + exponential backoff, ETag / Last-Modified disk cache with bulk revalidation, per-host throttle, cookie jar, basic + bearer auth, proxy, and three bulk concurrency models (parallel_fetch / multi_fetch / streaming multi_each). Scrapetor::Session ties the cookie / auth / throttle / retry policies together. Also ships robots.txt + sitemap.xml parsers, a bounded-memory streaming HTML parser, and structured-data extractors (JSON-LD, OpenGraph, Schema.org, Microdata, RDFa, Twitter Cards). The Net::HTTP-based Scrapetor.fetch is preserved as the no-libcurl fallback.
Total
Ranking: 197,231 of 198,774
Downloads: 329
Daily
Ranking: 103,943 of 198,739
Downloads: 0
Downloads Trends
Ranking Trends
Num of Versions Trends
Popular Versions (Major)
Popular Versions (Major.Minor)
Depended by
| Rank | Downloads | Name |
|---|
Depends on
| Rank | Downloads | Name |
|---|---|---|
| 8 | 1,469,998,038 | rake |
| 14 | 1,349,101,799 | minitest |
| 17 | 1,304,159,190 | nokogiri |
| 107 | 495,383,179 | webrick |
| 628 | 91,235,693 | benchmark-ips |
| 8,903 | 475,845 | nokolexbor |
Owners
| # | Gravatar | Handle |
|---|---|---|
| 1 | scrapetor |