ru_token

https://github.com/LoganBresnahan/ru_token

[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.