A transformer generates text one token at a time. Naively, producing token 100 would mean re-running the entire model over all 99 previous tokens from scratch — recomputing attention keys and values it already computed a moment ago. The KV cache is the fix: store each layer's keys and values as they're computed, and reuse them for every future token.
Without cache
- Recompute K,V for all past tokens every step
- Cost grows quadratically with length
- Wasteful, but conceptually simple
With cache
- Compute K,V once per token, store them
- Reuse for every future step
- Cost grows linearly with length
This is why LLM inference has two very different phases: prefill, where the whole prompt is processed at once (and the cache is built), and decode, where one new token is generated per step using the growing cache.
- The cache is the reason chat responses stream out token-by-token at a roughly steady rate instead of getting exponentially slower.
- The cache grows with every generated token and with every concurrent user — it's the dominant consumer of GPU memory during serving, often larger than the model's own weights.
- Techniques like grouped-query attention, quantized caches, and PagedAttention exist specifically to shrink or manage this memory.
Every serving system you'll encounter — vLLM, TensorRT-LLM, TGI — is fundamentally organized around managing this cache efficiently, because it, not the model's weights, is usually the binding memory constraint in production.