cjk-aware-text-metrics

What problem does it solve?

Fixed chars-per-token constants fail for Japanese/Chinese/Korean text, leading to underestimation of tokens and downstream costs and rate limits.

Core Features & Use Cases

• Detect CJK characters using Unicode ranges and compute weighted token counts that reflect multilingual content.
• Apply to multilingual LLM preprocessing, chunking, and cost estimation for mixed-language documents.
• Real-world use: process a Japanese document with mixed Latin text to produce accurate token counts for pricing and chunking.

Quick Start

Estimate token counts accurately for multilingual text by weighting CJK and Latin characters.