Sample prompt
from a prompt dataset
Policy generates response
current model, sampling
Reward model scores it
scalar reward r
Subtract KL to reference
penalize drift
PPO update
clip + advantage
Each PPO iteration is a loop over four models: the policy (being trained), the reference (frozen copy of the SFT model, used only to measure drift), the reward model (frozen, scores full responses), and a value model (predicts expected reward, used to reduce variance in the policy gradient).