the Bradley-Terry model

Think about how chess rankings work: every player has a rating, and the difference between two ratings predicts who will win. The Bradley-Terry model is the same idea applied to any kind of comparison. It is a simple, classic statistical recipe (from the 1950s) for turning a pile of 'A beat B' judgements into a single strength score for each option, so that the scores explain the wins you observed.

Here is the rule. Give each item a hidden strength s. The model says the probability that A is preferred over B is the logistic (sigmoid) function of the difference: p(A beats B) = 1 / (1 + e^-(s_A - s_B)). So if two answers have equal strength the probability is 0.5 (a coin flip); if A is one point stronger, A wins about 73% of the time; two points, about 88%. Training a reward model in RLHF is, in essence, choosing the strengths s (computed by the network) so these predicted probabilities best match the human comparisons collected.

This is the precise mathematical bridge from messy human preferences to the numerical reward that drives RLHF, and the same logistic form reappears inside direct preference optimization. It is worth knowing its assumptions, though: Bradley-Terry assumes a single consistent scale and treats each comparison as independent, so it can struggle when preferences are genuinely intransitive (people prefer A to B, B to C, yet C to A) or when raters systematically disagree. The numbers are a useful model of preference, not a law of it.