Prefix Caching
Prefix caching reuses the KV cache computed for a shared prompt prefix across multiple requests, skipping redundant prefill computation for identical system prompts or long common instructions.
What is Prefix Caching?
Prefix caching reuses the KV cache computed for a shared prompt prefix across multiple requests, skipping redundant prefill computation for identical system prompts or long common instructions.
Prefix caching reuses the KV cache computed for a shared prompt prefix across multiple requests, skipping redundant prefill computation for identical system prompts or long common instructions.
Where is it used?
vLLM's `enable_prefix_caching`, SGLang's RadixAttention, and Anthropic's prompt caching all exploit shared prefixes to cut TTFT for chat applications with repeated system prompts.
How to build it
Run two `generate()` calls with the same 500-token prefix and a different 10-token suffix; cache the prefix KV manually, reuse it via `past_key_values`, and measure the TTFT reduction for the second call.