What problem does it solve? Tokenizing large text corpora with pure Python implementations is slow, and building custom vocabularies for domain-specific or multilingual models requires deep knowledge of subword algorithms. This Skill provides guidance for using the Rust-based HuggingFace Tokenizers library to tokenize 1GB of text in under 20 seconds and train custom tokenizers from scratch. ## Core Features & Use Cases - High-Speed Tokenization: Encode text 10-100x faster than pure Python, with batch encoding, padding, truncation, and multi-processing support. - Custom Tokenizer Training: Train BPE, WordPiece, or Unigram tokenizers on files, iterators, or streaming datasets with configurable vocabulary size and special tokens. - Alignment Tracking: Map tokens back to original character offsets for NER, question answering, and token classification tasks. - Transformers Integration: Wrap custom tokenizers with PreTrainedTokenizerFast and use them with AutoTokenizer and any transformers model. - Use Case: Train a domain-specific BPE tokenizer on a medical corpus, wrap it for transformers, and use it to tokenize training data for a custom language model. ## Quick Start Train a BPE tokenizer with a 30,000-token vocabulary on my corpus file and show me how to encode text with padding and truncation.
