Reward Models · L4 Foundations

The statistical model behind reward-model training and the effect of miscalibration on downstream RL.

P (y_{w} ≻ y_{l} ∣ x) = \frac{exp ( r ^{*} ( x , y _{w} ))}{exp ( r ^{*} ( x , y _{w} )) + exp ( r ^{*} ( x , y _{l} ))}

This is the Bradley-Terry model in its original form — a special case of the Luce choice axiom for pairwise comparisons, historically used to rank chess and sports competitors from win/loss records. It assumes there exists some latent 'true strength' $r^{*} (x, y)$ for each response such that win probability is a logistic function of the strength difference. Reward-model training is exactly maximum-likelihood estimation of $r^{*}$ under this assumed generative model of human choice.