Token IDs
Token IDs are the integer indices assigned to each token in the vocabulary. They are the actual numbers fed into the embedding layer and predicted by the output softmax.
What is Token IDs?
Token IDs are the integer indices assigned to each token in the vocabulary. They are the actual numbers fed into the embedding layer and predicted by the output softmax.
Token IDs are the integer indices assigned to each token in the vocabulary. They are the actual numbers fed into the embedding layer and predicted by the output softmax.
Where is it used?
Every LLM forward pass converts text to token IDs, looks up embeddings by ID, and produces logits over IDs; the final layer maps hidden states back to vocabulary-size logits.
How to build it
Call `tokenizer.encode('Hello')` to get `[15496]`, pass it as `torch.tensor(ids)` to `model.generate()`, and decode the output IDs back with `tokenizer.decode`.