RLHF

Supervised fine-tuning (SFT) teaches a model to imitate demonstrations — it learns *what a good answer looks like* because you show it one. But for many qualities you care about — helpfulness, tact, refusing harmful requests without being useless — there is no single correct token sequence to imitate. There's only a comparison: this answer is better than that one. RLHF (Reinforcement Learning from Human Feedback) is the machinery for turning 'better than' judgments into weight updates.

• RL — the model (policy) takes actions (generates tokens) and gets a scalar reward, then updates to get more reward next time.
• Human feedback — the reward signal comes from people ranking model outputs, not from a hand-written reward function.
• PPO (Proximal Policy Optimization) — the specific RL algorithm almost everyone used to actually do the optimization.
• KL penalty — a leash that stops the model from wandering too far from its starting point while it chases reward.

This is the pipeline behind ChatGPT's original alignment recipe: a base model, then SFT on demonstrations, then a reward model trained on human preference rankings, then PPO to push the SFT model toward what the reward model likes — without letting it drift so far that it stops sounding like a coherent assistant.

• RLHF made models feel dramatically more usable — this is what turned GPT-3-style completion engines into ChatGPT-style assistants.
• It's also expensive: you need human raters, a reward model, and a finicky RL training loop with four models in memory at once.
• Because of that cost, cheaper alternatives like DPO (its own star) emerged and are now the default starting point for most teams.

Think of the whole pipeline as three separable pieces: (1) a way to score outputs, (2) an objective that rewards good scores while staying close to the original model, (3) an optimizer that actually climbs that objective. Later layers in this star unpack each piece.