One request at a time
- GPU mostly idle waiting on memory
- Great latency for that one user
- Terrible GPU utilization
Batched serving
- Many requests share one forward pass
- GPU stays busy doing matmuls
- Throughput goes up 10-100x
A single LLM forward pass for one user barely uses the GPU's compute — it's bottlenecked on reading the model's weights from memory, not on arithmetic. The fix is the same trick behind every high-throughput system: do the same work for many users at once. Batching means running several requests through the model in a single forward pass so the weight-reading cost is shared across all of them.
- Throughput — total tokens generated per second across all users.
- Latency — how long one particular user waits for their tokens.
- Continuous batching — new requests join and finished ones leave a batch on every step, instead of waiting for a fixed batch to fully complete.
- KV cache — the per-request memory that stores past attention keys/values so tokens aren't recomputed from scratch.
The catch: throughput and latency fight each other. Bigger batches mean better GPU utilization but each individual user's tokens may arrive slightly slower, and a very long request stuck in a batch can hold up shorter ones. Serving systems like vLLM, TensorRT-LLM, and TGI exist specifically to manage this tradeoff automatically.
- Naive batching (wait for a fixed group, run them together, wait for all to finish) wastes GPU time on requests that finish early.
- Continuous batching fixes this by swapping finished requests out and new ones in at every decoding step.
- Memory for the KV cache, not compute, is usually the true limit on how many requests can be served at once.