Long Context
Long-context refers to models and techniques designed to handle input sequences far exceeding the standard 2K-4K window, typically 32K to 2M tokens. This requires architectural or positional encoding modifications to maintain quality at scale.
What is Long Context?
Long-context refers to models and techniques designed to handle input sequences far exceeding the standard 2K-4K window, typically 32K to 2M tokens. This requires architectural or positional encoding modifications to maintain quality at scale.
Long-context refers to models and techniques designed to handle input sequences far exceeding the standard 2K-4K window, typically 32K to 2M tokens. This requires architectural or positional encoding modifications to maintain quality at scale.
Where is it used?
GPT-4 Turbo (128K), Claude 3 (200K), and Gemini 1.5 Pro (2M) are long-context models. Techniques include RoPE scaling (LLaMA-2 Long), YaRN, Ring Attention, and FlashAttention-2. Applications: entire-codebase reasoning, book summarization, long-document RAG.
How to build it
Extend RoPE with NTK-aware scaling: `base *= (scale * seq_len / original_max) ** (d / (d - 2))`. Use FlashAttention-2 to reduce memory from O(n²) to O(n). For extreme lengths, implement Ring Attention to split sequences across GPUs.