Categories: None [Edit]

tokenizer

https://rubygems.org/gems/tokenizer
https://github.com/arbox/tokenizer
A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

Total

Ranking: 10,364 of 186,698
Downloads: 258,265

Daily

Ranking: 5,451 of 186,685
Downloads: 137

Depended by

RankDownloadsName
8,351411,215metanorma-iso
70,43013,255social_tokenizer
80,90910,772jekyll-related-posts
98,4107,869smalltext
105,9976,840valkey-objects
138,2304,136shalmaneser-frappe
151,8183,513TacTalk
160,4772,973meiou
173,0602,144sentimentanalyzer

Depends on

RankDownloadsName

Owners

#GravatarHandle
1iconarbox