What problem does it solve? Tokenizing large text corpora with pure Python implementations is slow, and building custom vocabularies for new models or domains requires deep knowledge of subword algorithms. This Skill provides Rust-based tokenization that processes 1GB of text in under 20 seconds, plus complete guidance for training custom tokenizers from scratch. ## Core Features & Use Cases - Custom Tokenizer Training: Train BPE, WordPiece, or Unigram tokenizers on your own corpus with configurable vocabulary size, special tokens, and normalization pipelines. - High-Speed Batch Encoding: Tokenize large datasets with padding, truncation, and multi-processing, achieving 80x speedup over pure Python. - Alignment Tracking: Map tokens back to original character positions for NER, question answering, and token classification tasks. - Transformers Integration: Wrap custom tokenizers with PreTrainedTokenizerFast and use them directly with AutoTokenizer and any transformers model. - Use Case: You are pretraining a domain-specific language model on medical literature. Use this Skill to train a 50k BPE tokenizer on your PubMed corpus, configure BERT-style post-processing, and export it in transformers format for model training. ## Quick Start Train a custom BPE tokenizer with a 30,000 token vocabulary on my corpus file and show me how to encode text with it.
