The roofline model says a kernel's achievable throughput is capped by whichever is smaller: the hardware's peak compute (FLOP/s) or its peak memory bandwidth times the kernel's arithmetic intensity. A kernel is 'compute bound' if its intensity exceeds the hardware's compute-to-bandwidth ratio, and 'memory bound' otherwise.
Layer 4 · Foundations
The KV Cache
Arithmetic intensity, the roofline model, and deriving why decode is bandwidth-bound from first principles.
15 min read180 XP