Key Cache
The key cache stores the projected key vectors for all past tokens in each attention layer, avoiding recomputation of the K projection during autoregressive decoding. It is one half of the KV cache.
What is Key Cache?
The key cache stores the projected key vectors for all past tokens in each attention layer, avoiding recomputation of the K projection during autoregressive decoding. It is one half of the KV cache.
The key cache stores the projected key vectors for all past tokens in each attention layer, avoiding recomputation of the K projection during autoregressive decoding. It is one half of the KV cache.
Where is it used?
Every autoregressive transformer (GPT, Llama, Mistral) maintains a key cache per layer per head during generation; FlashDecoding and PagedAttention optimise its memory layout.
How to build it
In a custom attention layer, append `k` for the new token to a running `past_k` tensor with `torch.cat([past_k, k], dim=-2)` and pass it forward; print `.shape` each step to observe growth.