sentencepiece
CommunityMultilingual tokenization with SentencePiece.
Authorovachiever
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill covers language-independent tokenization with SentencePiece, supporting BPE and Unigram algorithms for multilingual NLP and CJK languages.
Core Features & Use Cases
- Language-Independent Tokenization: Train and apply tokenizers without language-specific rules.
- BPE & Unigram: Choose between deterministic (BPE) and probabilistic (Unigram) tokenization.
- Subword Regularization: Enable sampling to improve robustness.
Quick Start
Train a unigram model on a multilingual corpus and load it with a HuggingFace tokenizer.
Dependency Matrix
Required Modules
None requiredComponents
references
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: sentencepiece Download link: https://github.com/ovachiever/droid-tings/archive/main.zip#sentencepiece Please download this .zip file, extract it, and install it in the .claude/skills/ directory.