skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

sentencepiece

Train and apply language-independent BPE and Unigram tokenizers on raw Unicode text.

1|Updated Mar 5, 2026
Search Tags:#nlp#multilingual#tokenization#sentencepiece#bpe#unigram#subword
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill sentencepiece-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sentencepiece
Source: https://github.com/Clay-HHK/claude-skills/tree/main/sentencepiece
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill sentencepiece-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sentencepiece, transformers, and includes references (resource) components.

What problem does it solve? Tokenizing multilingual or CJK text usually requires language-specific pre-processing rules that are brittle and hard to reproduce. SentencePiece trains tokenizers directly on raw text, treating whitespace as a symbol, so one deterministic vocabulary works across all languages. ## Core Features & Use Cases - BPE and Unigram Training: Train subword vocabularies from raw corpora via the command line or Python API, with configurable vocab size, character coverage, and special tokens. - Deterministic Encoding and Decoding: Encode text to pieces or IDs and decode back losslessly at roughly 50,000 sentences per second with about 6MB memory. - Subword Regularization: Sample alternative tokenizations during training for data augmentation and model robustness. - Use Case: Train a 32k Unigram tokenizer on a multilingual corpus with T5-style sentinel tokens, then load it through the transformers T5Tokenizer for model pretraining. ## Quick Start Train a SentencePiece Unigram tokenizer with 8000 vocabulary on my corpus.txt file and show me how to encode and decode a sample sentence with it.

Frequently Asked Questions about sentencepiece

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a SentencePiece tokenizer in Python?▼

Call spm.SentencePieceTrainer.train with input corpus path, model_prefix, vocab_size, and model_type set to 'unigram' or 'bpe'. This produces a .model binary and .vocab file you load with SentencePieceProcessor for encoding.

SentencePiece BPE vs Unigram: which should I use?▼

Unigram is probabilistic, supports sampling for subword regularization, and suits multilingual models like T5 and XLNet. BPE trains faster and is deterministic, used by mBART. Choose Unigram for most multilingual tasks.

Does SentencePiece work with Chinese and Japanese text?▼

Yes, SentencePiece is language-independent and handles CJK text without pre-segmentation. Set character_coverage to 1.0 when training on Chinese, Japanese, or Korean corpora to cover all characters.

Can I use SentencePiece tokenizers with HuggingFace transformers?▼

Yes, models like T5, ALBERT, XLNet, and mBART use SentencePiece internally. Load them via classes like T5Tokenizer.from_pretrained, which wraps the SentencePiece model automatically.

When should I use HuggingFace Tokenizers instead of SentencePiece?▼

HuggingFace Tokenizers offers faster training and tokenization (about 4x faster at 200k sentences/sec) and more flexibility. SentencePiece is preferable when you need deterministic reproducible vocabularies or T5-style training on raw text.

Related Skills

Explore More Education & Research Skills→
Moshe-shipMoshe-ship
29

safha

Collect and clean Arabic web content into JSONL training data.

Community
Intermediate
NzettodessNzettodess

sentencepiece

Train and apply language-independent BPE or Unigram tokenizers on raw Unicode text.

Community
Intermediate
tae2089tae2089
2

recipe-agent-team-terminology-context

Generate terminology.md and context-map.md to align project vocabulary.

Community
Advanced
eyadsibaieyadsibai
7

huggingface-tokenizers

Trains custom BPE and WordPiece tokenizers using HuggingFace tokenizers library.

Community
Intermediate