Bradley–Terry model

Suppose every option has a hidden 'strength' — a chess player's skill, an answer's quality. The Bradley-Terry model is a simple, century-old rule for turning those hidden strengths into the probability that one beats the other head-to-head. The bigger the gap in strength, the more lopsided the expected outcome; equal strengths give a coin flip. It is the bridge from 'how good is each thing' to 'which one will a person pick'.

Assign each item a real score s_i. Bradley-Terry says the probability that i is preferred to j is the logistic of their difference, σ(s_i − s_j). In RLHF those scores are exactly the reward model's outputs, so fitting the reward model to human comparisons is maximum-likelihood estimation under this model — which is why the standard reward loss is a sigmoid cross-entropy on score differences. The whole preference-to-reward step rests on this one assumption.

The model assumes preferences are transitive and stochastically consistent — that a single scalar strength explains every comparison. Real human preferences violate this with cycles, context effects, and raters who simply disagree, so Bradley-Terry is a tractable and useful approximation, not the truth. It also fixes only differences of scores: there is no absolute zero, so the reward it produces is meaningful only up to a constant shift.