huggingface-tokenizers

Initialize a Tokenizer with the BPE model, set a pre-tokenizer like Whitespace or ByteLevel, then call train() with your corpus files and a BpeTrainer configured with vocab_size and special_tokens. Training 100MB of text takes roughly 1-2 minutes on a modern CPU.

BPE merges the most frequent token pairs and is used by GPT-2 and RoBERTa. WordPiece scores merges by likelihood ratio and powers BERT. Unigram starts from a large vocabulary and prunes tokens probabilistically, suiting multilingual models like T5 and ALBERT.

Save your trained tokenizer to JSON, then wrap it with PreTrainedTokenizerFast specifying unk_token, pad_token, and other special tokens. After save_pretrained(), it loads via AutoTokenizer and works with any transformers model.

Yes, fast tokenizers return offset mappings that link each token to character positions in the original text. Use output.offsets, word_ids(), and char_to_token() to align predictions back to source spans for NER and question answering.

A high [UNK] rate usually means the vocabulary is too small or min_frequency is too high for your corpus. Increase vocab_size, lower min_frequency, or switch to byte-level BPE, which can represent any character and eliminates unknown tokens entirely.

Use tiktoken when matching OpenAI GPT model tokenization exactly, and SentencePiece when replicating T5 or ALBERT training pipelines. For training custom tokenizers or production NLP pipelines, the tokenizers library offers faster Rust-based performance.