Incoming request queue
Scheduler (admit/evict per step)
Active decode batch
Paged KV cache allocator
Real serving engines (vLLM, TensorRT-LLM, TGI) are large C++/CUDA systems, but the scheduling logic is simple enough to prototype in pure Python. Below is a minimal continuous-batching loop that admits new requests, runs one decode step for everyone active, and evicts finished ones — the same shape as the real thing, minus the GPU kernels.