RLHF (PPO)
- Train a reward model
- Sample rollouts from policy
- Run RL (PPO) against reward model
- 4 models in memory
DPO
- No reward model
- No sampling/rollouts
- One supervised-style loss on preference pairs
- 2 models in memory
Direct Preference Optimization (DPO) achieves the same goal as RLHF — a model that prefers what humans prefer — with a training recipe that looks almost exactly like ordinary supervised fine-tuning. You feed it the same kind of preference pairs (prompt, better response, worse response) that would train a reward model, but there is no reward model and no reinforcement learning loop at all.
Practically, this means DPO training looks like: take a batch of (prompt, chosen, rejected) triples, compute how much more likely the model finds the chosen response versus the rejected one (relative to a frozen reference copy), and push that ratio in the right direction with a simple loss. No sampling from the model during training, no reward-model forward passes, no PPO instability.
- DPO needs the exact same preference data as reward-model training — nothing new to collect.
- It trains like supervised learning: stable, single loss, standard optimizers, no rollout generation loop.
- It has become the default starting point for preference optimization at most labs and open-source projects because it's dramatically simpler to get right.
The trade-off: DPO's quality ceiling is set entirely by the fixed offline dataset of preference pairs it's trained on, whereas PPO-based RLHF can keep sampling *new* responses from the current (improving) policy and getting them freshly scored — an 'online' advantage that matters most when you have a very reliable reward signal to sample against.