Speculative Decoding · L2 Mechanics

Draft model samples k tokens
fast, sequential, cheap
Target model scores all k+1 positions
one parallel forward pass
Accept/reject each draft token
using probability ratio
Resample at first rejection
from a corrected distribution

One speculative decoding round.

Let the draft model's distribution at position $t$ be $q (x)$ and the target model's be $p (x)$ . For each drafted token $x_{t} \sim q$ , accept it with probability $min (1, p (x_{t}) / q (x_{t}))$ . If accepted, move to the next drafted token. If rejected, discard the rest of the draft and sample a replacement token from a corrected 'leftover' distribution instead of from $p$ directly.