Value Cache
The value cache stores projected value vectors for past tokens so the attention output can be computed without re-running the V projection on prior positions. It pairs with the key cache.
What is Value Cache?
The value cache stores projected value vectors for past tokens so the attention output can be computed without re-running the V projection on prior positions. It pairs with the key cache.
The value cache stores projected value vectors for past tokens so the attention output can be computed without re-running the V projection on prior positions. It pairs with the key cache.
Where is it used?
Used alongside the key cache in every autoregressive generation path of HF Transformers, vLLM, and TGI; quantised value caches (fp8/int8) are used inllama.cpp and TensorRT-LLM to save memory.
How to build it
Maintain `past_v` per layer, concatenate the new value with `torch.cat([past_v, v], dim=-2)`, compute attention as `softmax(q @ past_k.T) @ past_v`, and verify output matches uncached generation.