reward model

A reward model is a stand-in for a human judge. You cannot have a person grade every one of the millions of responses an RL run produces, so you train a model to predict that grade: given a prompt and a response, it outputs a single number meant to capture how much a person would prefer that response. It is usually built from the same kind of transformer as the policy, with the final layer replaced by one that emits a scalar score.

It learns from preference pairs using the Bradley-Terry model: the probability that one response beats another is the logistic function of the difference in their scores. Training nudges the reward of chosen responses above rejected ones, so the model absorbs the pattern of human taste rather than memorizing any single answer.

The catch is that the reward model is a proxy, not the real thing. It is accurate near the data it was trained on but can be wildly wrong on the strange, off-distribution outputs that an optimizer will happily hunt for, which is the seed of reward hacking and over-optimization.