Vocabulary
The vocabulary is the fixed set of all tokens a tokenizer can produce, each mapped to a unique integer ID. It defines the input and output space of the language model.
What is Vocabulary?
The vocabulary is the fixed set of all tokens a tokenizer can produce, each mapped to a unique integer ID. It defines the input and output space of the language model.
The vocabulary is the fixed set of all tokens a tokenizer can produce, each mapped to a unique integer ID. It defines the input and output space of the language model.
Where is it used?
GPT-4 has a ~100k vocabulary; Llama 3 has 128k tokens; a larger vocabulary covers more languages and rare words but increases the embedding and output projection parameter count.
How to build it
Load any tokenizer with `AutoTokenizer.from_pretrained('gpt2')`, print `len(tokenizer)` and `tokenizer.get_vocab()`, and observe how vocabulary size ties to embedding matrix shape.