Prefill
process the whole prompt at once
Populate KV cache
one entry per prompt token
Decode loop
generate one token per step
Append new KV entries
cache grows by one per step
Serving splits into two phases with opposite bottlenecks. Prefill processes the entire prompt in one parallel forward pass — it's compute-bound, since all prompt tokens' matmuls can run together. Decode generates one new token at a time, each step only computing for a single new token per request — it's memory-bound, since the whole KV cache and weight set must be read for very little new arithmetic.