JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Turning preferences into a reward model

A pile of "A beats B" labels isn't a reward yet. The Bradley-Terry model is the bridge that turns comparisons back into a single number you can optimize.

Why we need a model at all

Comparisons only cover the responses raters actually saw. But during RL the policy will produce millions of new responses no human ever judged. We need something that can score those too — a reward model learned from preferences: a model that takes any response and outputs a scalar standing in for "how much a human would like this".

In language settings this is the reward model: usually a copy of the base model with its final layer swapped for a single output number. It has read the same text the policy has, so it understands the responses it must grade.

The reward model is just a network that reads a response and emits one number — the base model with its final layer swapped for a single scalar output.

A multilayer neural network: inputs flow through hidden layers to a single output node, representing the reward model's scalar score.

The Bradley-Terry bridge

How do you train a model that outputs a number using data that only contains winners? The Bradley-Terry model is the classic answer. It assumes each item has a hidden "strength", and the chance that A beats B grows smoothly as A's strength exceeds B's. Concretely, the probability A wins is the logistic (sigmoid) of the difference between their two reward scores.

P(i \succ j) = \sigma\!\left(r_i - r_j\right) = \dfrac{e^{r_i}}{e^{r_i} + e^{r_j}}

The Bradley-Terry bridge: the chance item i beats j is a sigmoid of their hidden reward difference.

That single assumption is the magic. It lets us define a loss directly on comparisons: make the winner's predicted reward higher than the loser's, just enough to explain the observed choices. Minimize that loss and out pops a reward function — even though no example was ever labelled with an absolute score.

# r = reward model; chosen/rejected = the pair, chosen preferred
import torch.nn.functional as F

def bt_loss(r, chosen, rejected):
    # P(chosen wins) = sigmoid(r(chosen) - r(rejected))
    margin = r(chosen) - r(rejected)
    return -F.logsigmoid(margin).mean()   # push the margin positive
Bradley-Terry loss: reward the winner above the loser, by a margin the data supports.

Training and sanity-checking it

Reward model training is then ordinary supervised learning on the comparison dataset. The metric you care about is preference accuracy: on held-out pairs, how often does the model give the chosen response the higher reward? Anything near random means the signal isn't there yet.

Reward-model training is just the supervised loop run on the comparison dataset.

Diagram of the supervised-learning loop: data, model, prediction, loss, update.

Don't trust just one

A single reward model has blind spots, and the policy will find them (the next guides are largely about this). A cheap, effective defence is reward model ensembles: train several reward models on different data shuffles or seeds and combine them. Where they agree, trust the reward; where they disagree, the score is uncertain — a useful flag for the optimizer to back off.

Grading the answer vs grading the work

So far the reward judges a finished response. But for multi-step tasks — solving a maths problem, a long plan — you often want to reward each step, not just the final answer. That is the difference between process and outcome rewards, and a process reward model scores the reasoning along the way.

Process rewards give denser feedback and make credit assignment easier — the policy learns which step helped — but they need labels on intermediate steps, which are harder to collect. It's a real trade-off, and which you choose depends on how much your task hides its quality inside the steps.