Categories: None [Edit]

tokenizer-ruby

https://rubygems.org/gems/tokenizer-ruby
https://github.com/johannesdwicahyo/tokenizer-ruby
Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.

Total

Ranking: 178,331 of 198,575
Downloads: 2,523

Daily

Ranking: 21,048 of 198,547
Downloads: 7

Depended by

RankDownloadsName
195,509485ner-ruby

Depends on

RankDownloadsName
1,29238,005,421rb_sys

Owners

#GravatarHandle
1iconjohannesdwicahyo