LLM AtlasObserver · 0 XP
Pricing
RLHF

Layer 4 · Foundations

RLHF

The KL-constrained RL objective in full, why it has a closed-form optimal policy, and PPO's variance-reduction math.

15 min read180 XP

  1. Constrained objective

    max reward, bounded KL

  2. Lagrangian form

    reward minus beta*KL

  3. Closed-form optimum

    Gibbs / softmax policy

From the constrained problem to its analytic solution — this is the derivation DPO exploits.

The RLHF objective is a KL-regularized reward maximization over the whole space of possible policies :