Batching & Serving · L4 Foundations

The roofline model, arithmetic intensity, and why decode throughput has a hard memory-bandwidth ceiling.

Prefill FLOPs/bytecompute-bound region

Decode FLOPs/bytememory-bound region

GPU ridge pointhardware's compute:bandwidth ratio

Arithmetic intensity of prefill vs. decode relative to a GPU's ridge point (illustrative numbers).

The roofline model describes any kernel's achievable throughput as the minimum of two limits: peak compute (FLOP/s) and peak memory bandwidth translated into FLOPs via the kernel's *arithmetic intensity* — how many floating-point operations it does per byte moved from memory.