Caching
Caching in LLM systems stores computed KV caches (prefix caching), embeddings, or full responses to skip redundant computation for repeated prompts and reduce latency.
What is Caching?
Caching in LLM systems stores computed KV caches (prefix caching), embeddings, or full responses to skip redundant computation for repeated prompts and reduce latency.
Caching in LLM systems stores computed KV caches (prefix caching), embeddings, or full responses to skip redundant computation for repeated prompts and reduce latency.
Where is it used?
vLLM `--enable-prefix-caching`, GPTCache, and Redis-based response caches are common; Anthropic's prompt caching bills less for repeated long prefixes by reusing KV state.
How to build it
Enable vLLM prefix caching with `--enable-prefix-caching`, benchmark two identical-prompt-prefix requests, and measure the TTFT drop; add a Redis response cache for exact-match prompts.