EOS Token
The EOS (end-of-sequence) token is a special vocabulary entry that signals the model has finished generating. When the model emits EOS, the generation loop stops and the final sequence is returned.
What is EOS Token?
The EOS (end-of-sequence) token is a special vocabulary entry that signals the model has finished generating. When the model emits EOS, the generation loop stops and the final sequence is returned.
The EOS (end-of-sequence) token is a special vocabulary entry that signals the model has finished generating. When the model emits EOS, the generation loop stops and the final sequence is returned.
Where is it used?
Every LLM uses an EOS token: `<|endoftext|>` in GPT-2, `</s>` in T5/BART, `<eos>` in LLaMA. Special tokens like `<|im_end|>` in ChatML serve as turn-ending equivalents. Tokenizers map these to fixed integer IDs.
How to build it
Get the EOS ID from the tokenizer: `eos_id = tokenizer.eos_token_id`. In the generation loop: `if next_token.item() == eos_id: break`. Append EOS during training so the model learns to emit it.