KV Cache Memory
KV-cache memory is VRAM used to store past key and value tensors for every attention layer so that previously generated tokens need not be recomputed. It grows linearly with sequence length and batch size.
What is KV Cache Memory?
KV-cache memory is VRAM used to store past key and value tensors for every attention layer so that previously generated tokens need not be recomputed. It grows linearly with sequence length and batch size.
KV-cache memory is VRAM used to store past key and value tensors for every attention layer so that previously generated tokens need not be recomputed. It grows linearly with sequence length and batch size.
Where is it used?
vLLM, TensorRT-LLM, and Hugging Face `generate()` with `use_cache=True` maintain KV caches; Llama-3 70B inference can consume tens of GB of KV cache for long contexts, driving PagedAttention optimization.
How to build it
Load a small GPT-2 with `use_cache=True`, generate 100 tokens, and monitor `torch.cuda.memory_allocated()` growth; inspect `past_key_values` tuple shapes to see per-layer K/V tensor dimensions.