A reward model (RM) takes a prompt and a response and outputs a single number: how good is this response? It's not a chatbot — it never generates text. It's a critic, trained purely to reproduce human judgments about which of two answers is better.
Language model
- Input: prompt
- Output: next token distribution
- Trained on: next-token prediction
Reward model
- Input: prompt + full response
- Output: one scalar score
- Trained on: pairwise human preferences
Reward models are almost always built by taking a pretrained (often SFT) language model and replacing its output head with a single linear layer that produces one number instead of a distribution over the vocabulary. The transformer backbone already understands language; the new head just needs to learn to map that understanding onto 'humans would rate this highly'.
The RM is the load-bearing piece of the whole RLHF pipeline: PPO can only ever be as good as the reward signal it's climbing. A reward model that's miscalibrated, biased toward length, or easily fooled will produce a policy that exploits exactly those flaws — this is the essence of reward hacking.
- Reward models are trained once (or a few times) and then frozen while the policy trains against them.
- They generalize surprisingly well within the distribution of prompts they were trained on, and poorly outside it.
- Modern practice often mixes rule-based/verifiable reward (unit tests passing, math checkers) with a learned RM for anything that isn't automatically checkable.