Subword Tokenization
Subword tokenization splits text into units between characters and words — common words stay whole, rare words break into meaningful pieces ('unfriend' → 'un', 'friend'). It balances vocabulary size and sequence length.
What is Subword Tokenization?
Subword tokenization splits text into units between characters and words — common words stay whole, rare words break into meaningful pieces ('unfriend' → 'un', 'friend'). It balances vocabulary size and sequence length.
Subword tokenization splits text into units between characters and words — common words stay whole, rare words break into meaningful pieces ('unfriend' → 'un', 'friend'). It balances vocabulary size and sequence length.
Where is it used?
Subword tokenization is the standard for all modern LLMs — BPE (GPT), WordPiece (BERT), and SentencePiece (Llama, T5) are all subword methods.
How to build it
Train a tokenizer with `tokenizers.Tokenizer(models.BPE())` from HuggingFace, feed it a corpus, and inspect how it splits a rare word into subwords versus keeping common words intact.