ruby_llm-tokenizer

https://github.com/washu/ruby_llm-tokenizer

Pure-Ruby facade over Hugging Face `tokenizers`, OpenAI `tiktoken_ruby`, and SentencePiece bindings that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).