Autoregressive decoding generates one token at a time, and each token requires a full pass through the (often huge) model. That's slow, and it's slow specifically because generation is *sequential* — you can't compute token 5 until you know token 4. Speculative decoding is a clever way to break that sequential bottleneck without changing a single output token.
- Draft model — a small, cheap model that quickly guesses several upcoming tokens.
- Target model — the large, expensive model whose outputs you actually want.
- Verification — the target model checks the draft's guesses in one batched pass, accepting the ones that match what it would have produced itself.
- Lossless — done correctly, the final output distribution is mathematically identical to just running the target model alone.
The magic is that verifying draft tokens with the target model costs about the same as generating *one* target token, because verification is a single parallel forward pass over positions — and modern GPUs are much better at doing tokens' worth of compute in parallel than doing 1 token, times in sequence, since decode is memory-bound (see the batching-and-serving star).
- Best case: the draft model guesses correctly for several tokens in a row, and you get multiple tokens for the price of one target forward pass.
- Worst case: the draft is wrong immediately, you fall back to the target model's own token, and you're no worse off than normal decoding.
- The speedup depends entirely on how well the small model predicts the big model's behavior — similar training data and vocabulary help a lot.