reward model (from preferences)

Humans cannot hand-label the millions of examples an RL agent will generate, but you can collect a few thousand comparisons and learn to imitate the human judge. A reward model is a network trained to do exactly that: read a candidate behaviour and output a single number — roughly how much a person would like it. Once trained, it scores the endless stream of new behaviours the agent produces, standing in for the human who is no longer in the room.

Usually the reward model shares the policy's architecture — for language models, the same transformer — with a scalar output head bolted on. It is trained on pairs of (preferred, rejected) outputs: the loss rewards it for assigning a higher score to the chosen item, typically through the Bradley-Terry log-likelihood. The result is a differentiable, queryable reward field r_φ that an RL optimiser can call as often as it likes, which is precisely what the next stage needs.

The reward model is both the lynchpin and the weak point of RLHF. It is only as good as the comparisons it saw, it grows overconfident outside their distribution, and the policy will relentlessly seek out wherever it is mistaken. That last tendency — the optimiser drawn to the reward model's errors — is the core danger known as reward model overoptimisation.