Inference Optimization
Batching & Serving
Continuous batching, PagedAttention, and throughput economics.
You can jump straight in, but this star assumes The KV Cache. Not recommended as a first stop.
Four layers of depth
Each layer ends with a quiz. Finish layer 4 and you own this concept.
- L1IntuitionWhy serving an LLM to many users at once is a completely different problem than running one prompt.5m
40 XP - L2MechanicsThe mechanics of continuous batching, paged KV cache memory, and the prefill/decode split.8m
70 XP - L3CodeImplement a toy continuous-batching scheduler and see how a paged KV cache allocator works.12m
110 XP - L4FoundationsThe roofline model, arithmetic intensity, and why decode throughput has a hard memory-bandwidth ceiling.15m
180 XP
33 stars in the atlas.