Direct Preference Optimization · L4 Foundations

The full derivation from the KL-constrained RLHF objective to the DPO loss, and its gradient's implicit reward-model interpretation.

π_{θ} max E_{y \sim π_{θ}} [r (x, y)] - β KL (π_{θ} ∥ π_{ref}) ⟹ π_{r} (y ∣ x) = \frac{1}{Z ( x )} π_{ref} (y ∣ x) exp (\frac{1}{β} r (x, y))

This is the same closed-form result from the RLHF star: for any fixed reward $r$ , the KL-constrained optimum $π_{r}$ is a reweighting of the reference by exponentiated reward. DPO's derivation inverts this relationship — treating it as defining $r$ as a function of $π_{r}$ and $π_{ref}$ :