twpipeline

https://github.com/taiwan-corpora/twpipeline

Nine JSONL stages from a raw web crawl to dispersion-corrected frequency and n-gram tables: source gating, Taiwan-standard punctuation normalization, sentence segmentation, character-inventory and lexical-origin filtering, block-level evidence judgment, type deduplication, Viterbi word segmentation and external-sort counting. Resource-bounded by declared cores and memory.