Maximum Tokens
The maximum tokens limit caps the number of tokens the model may generate in a single response. It bounds inference cost and latency, ensuring the generation loop terminates even without an EOS token.
What is Maximum Tokens?
The maximum tokens limit caps the number of tokens the model may generate in a single response. It bounds inference cost and latency, ensuring the generation loop terminates even without an EOS token.
The maximum tokens limit caps the number of tokens the model may generate in a single response. It bounds inference cost and latency, ensuring the generation loop terminates even without an EOS token.
Where is it used?
Every LLM API exposes `max_tokens` (OpenAI) or `max_new_tokens` (HuggingFace). GPT-4 Turbo allows up to 4096 output tokens; Claude 3 allows 4096. Setting it too low truncates answers; too high wastes compute and time.
How to build it
Track `generated_count` in the loop: `for i in range(max_new_tokens): ...; if i + 1 >= max_new_tokens: break`. Alternatively check `if ids.shape[1] - prompt_len >= max_new_tokens: break`.