Maximum Context Length
The maximum context length is the total number of tokens (prompt + generated) a model can process in one forward pass. It is fixed by the model's architecture (positional embedding limit) and determines the longest conversation or document it can handle.
What is Maximum Context Length?
The maximum context length is the total number of tokens (prompt + generated) a model can process in one forward pass. It is fixed by the model's architecture (positional embedding limit) and determines the longest conversation or document it can handle.
The maximum context length is the total number of tokens (prompt + generated) a model can process in one forward pass. It is fixed by the model's architecture (positional embedding limit) and determines the longest conversation or document it can handle.
Where is it used?
GPT-3 had 2K, GPT-4 Turbo supports 128K, Claude 3 supports 200K, and Gemini 1.5 Pro supports 2M tokens. Exceeding this limit causes errors or silent truncation. Long-context models use RoPE or ALiBi to extrapolate position embeddings.
How to build it
Before generation, check `if len(prompt_ids) + max_new_tokens <= model_max_len: proceed else: truncate or error`. Use `tokenizer.model_max_length` to read the limit. Implement sliding-window attention or chunking for long inputs.