Post-training & Alignment
Direct Preference Optimization
Preference alignment without the reinforcement learning.
You can jump straight in, but this star assumes RLHF. Not recommended as a first stop.
Four layers of depth
Each layer ends with a quiz. Finish layer 4 and you own this concept.
- L1IntuitionDPO in one idea: train directly on preference pairs, no reward model or RL loop required.5m
40 XP - L2MechanicsThe mechanics of the DPO loss: log-probability ratios, the implicit reward, and how the objective is derived conceptually.8m
70 XP - L3CodeImplement the DPO loss and a full training step end to end, including reference log-prob caching.12m
110 XP - L4FoundationsThe full derivation from the KL-constrained RLHF objective to the DPO loss, and its gradient's implicit reward-model interpretation.15m
180 XP
33 stars in the atlas.