Direct Preference Optimization · L2 Mechanics

The mechanics of the DPO loss: log-probability ratios, the implicit reward, and how the objective is derived conceptually.

Prompt + chosen + rejected
Score both under policy
Score both under frozen reference
Compare the two log-ratios
Push chosen's advantage up

What actually happens inside one DPO training step.

Recall from the RLHF derivation that the optimal KL-constrained policy has the form $π^{*} (y ∣ x) \propto π_{ref} (y ∣ x) exp (r (x, y) / β)$ . Solving this for $r (x, y)$ gives an *implicit reward* defined entirely by how much more likely the policy makes a response than the reference does: