SentencePiece
SentencePiece is a language-independent subword tokenizer that treats the input as a raw byte stream, so it works on any language without pre-tokenization on whitespace. It supports BPE and Unigram modes.
What is SentencePiece?
SentencePiece is a language-independent subword tokenizer that treats the input as a raw byte stream, so it works on any language without pre-tokenization on whitespace. It supports BPE and Unigram modes.
SentencePiece is a language-independent subword tokenizer that treats the input as a raw byte stream, so it works on any language without pre-tokenization on whitespace. It supports BPE and Unigram modes.
Where is it used?
SentencePiece is used by Llama 2/3, T5, ALBERT, and multilingual models like XLM-R; its whitespace-agnostic design is crucial for languages without spaces (Chinese, Japanese).
How to build it
Install `sentencepiece`, train with `spm.SentencePieceTrainer.train('--input=corpus.txt --model_prefix=m --vocab_size=32000'`, and encode/decode with `spm.SentencePieceProcessor`.