Tokenization
Text to tokens to IDs. The first step in every LLM pipeline. BPE, WordPiece, and vocabulary design.
Learning path
Work through each subtopic in order. Click any file below to open its lesson. Track your progress with the checkbox at the bottom.
Subtopics
Each subtopic includes a concise explanation, code examples, and references.
Tokenization
Text → tokens interactive viewer.
Character Tokenization
Character sequence.
Word Tokenization
Word segmentation viewer.
Subword Tokenization
Token-merging animation.
BPE
BPE training playground.
WordPiece
Subword segmentation viewer.
SentencePiece
Tokenization comparison playground.
Vocabulary
Searchable vocabulary table.
Token IDs
Token ↔ ID mapping.
Special Tokens
Special-token control panel.
Vocabulary Size
Vocabulary-size slider.
Token Count
Live token counter.