Special Tokens
Special tokens are reserved non-text tokens like `<bos>`, `<eos>`, `<pad>`, and `<unk>` that signal sequence boundaries, padding, or unknown words. They are part of the vocabulary but not natural language.
What is Special Tokens?
Special tokens are reserved non-text tokens like `<bos>`, `<eos>`, `<pad>`, and `<unk>` that signal sequence boundaries, padding, or unknown words. They are part of the vocabulary but not natural language.
Special tokens are reserved non-text tokens like `<bos>`, `<eos>`, `<pad>`, and `<unk>` that signal sequence boundaries, padding, or unknown words. They are part of the vocabulary but not natural language.
Where is it used?
Llama uses `<bos>` to start sequences; GPT-2 uses `<|endoftext|>` as document separator; padding tokens batch variable-length sequences in training and inference.
How to build it
Inspect `tokenizer.special_tokens_map` and `tokenizer.pad_token_id`, encode a pair of sequences with `padding=True`, and observe where pad tokens appear in the ID tensor.