Vocabulary Size
Vocabulary size is the number of distinct tokens the tokenizer supports. It directly determines the embedding matrix and final output projection dimensions, so it is a key architectural tradeoff.
What is Vocabulary Size?
Vocabulary size is the number of distinct tokens the tokenizer supports. It directly determines the embedding matrix and final output projection dimensions, so it is a key architectural tradeoff.
Vocabulary size is the number of distinct tokens the tokenizer supports. It directly determines the embedding matrix and final output projection dimensions, so it is a key architectural tradeoff.
Where is it used?
Smaller models use 32k vocab (Llama 2); larger multilingual models use 128k (Llama 3) or 256k (Qwen2) to cover more languages without excessive sequence lengths.
How to build it
Compare `nn.Embedding(32000, 4096)` vs `nn.Embedding(128000, 4096)` parameter counts with `sum(p.numel() for p in layer.parameters())` to see the memory cost of larger vocab.