Context Limit
The context limit is the hard upper bound on tokens a model accepts, enforced by the API or inference engine. Inputs exceeding it are rejected, truncated, or cause an error. It is distinct from the context window (architectural max) and may be set lower.
What is Context Limit?
The context limit is the hard upper bound on tokens a model accepts, enforced by the API or inference engine. Inputs exceeding it are rejected, truncated, or cause an error. It is distinct from the context window (architectural max) and may be set lower.
The context limit is the hard upper bound on tokens a model accepts, enforced by the API or inference engine. Inputs exceeding it are rejected, truncated, or cause an error. It is distinct from the context window (architectural max) and may be set lower.
Where is it used?
OpenAI enforces context limits per model (e.g., 8192 for `gpt-4`, 128K for `gpt-4-turbo`). HuggingFace TGI and vLLM set `--max-input-length` to cap prompts. Exceeding it returns a 400 error or silently truncates from the left.
How to build it
Validate before sending: `if len(prompt_ids) + max_new > limit: raise ValueError('Context exceeded')`. For graceful handling, truncate from the left: `prompt_ids = prompt_ids[:, -limit:]` or implement a retrieval-augmented summarization step.