Prompt tokens arrive
Prefill: parallel forward pass
compute-bound
Cache populated
Decode: one token per step
memory-bandwidth-bound
Cache appended each step
| Phase | Parallelism | Bottleneck | What grows |
|---|---|---|---|
| Prefill | All prompt tokens at once | Compute (large matmuls) | Cache filled once |
| Decode | One token at a time | Memory bandwidth (weights reread every step) | Cache grows by one slot per step |