Tokenizer
A tokenizer converts raw text into a sequence of integer token IDs that the model can process. For a small LLM, this is often a character-level or simple BPE tokenizer trained on the dataset's vocabulary, mapping each unit to a unique ID.
What is Tokenizer?
A tokenizer converts raw text into a sequence of integer token IDs that the model can process. For a small LLM, this is often a character-level or simple BPE tokenizer trained on the dataset's vocabulary, mapping each unit to a unique ID.
A tokenizer converts raw text into a sequence of integer token IDs that the model can process. For a small LLM, this is often a character-level or simple BPE tokenizer trained on the dataset's vocabulary, mapping each unit to a unique ID.
Where is it used?
GPT-2 uses BPE with 50K tokens; LLaMA uses SentencePiece with 32K. For educational small LLMs, Karpathy's nanoGPT trains a custom BPE tokenizer with `tiktoken` or uses a character-level tokenizer for the Shakespeare dataset (vocab size ~65).
How to build it
Character-level: build `chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}; encode = lambda s: [stoi[c] for c in s]`. BPE: `from tiktoken import get_encoding; enc = get_encoding('gpt2')` or train custom with `tokenizers` library.