Cache Growth
Cache growth is the linear increase in KV-cache memory and the quadratic increase in attention compute as sequence length grows during generation, eventually bottlenecking throughput and context length.
What is Cache Growth?
Cache growth is the linear increase in KV-cache memory and the quadratic increase in attention compute as sequence length grows during generation, eventually bottlenecking throughput and context length.
Cache growth is the linear increase in KV-cache memory and the quadratic increase in attention compute as sequence length grows during generation, eventually bottlenecking throughput and context length.
Where is it used?
Long-context models like Claude (200k tokens) and Gemini (1M tokens) fight cache growth with sliding-window attention, cache compression, and quantised KV; vLLM tracks cache pressure to schedule batches.
How to build it
Generate 512 tokens with GPT-2 and log `past_key_values[0][0].shape` each step; plot the last-dimension size to visualise linear growth and measure VRAM at intervals with `torch.cuda.memory_allocated()`.