a reward model

Think of a judge at a baking contest who has tasted thousands of cakes alongside the head chef and learned to predict the chef's verdict. The judge cannot bake, but given any new cake she can give it a score that usually agrees with what the chef would say. A reward model plays exactly this role in RLHF: it is a separate model whose only job is to look at an answer and output a single number, a reward, estimating how much a human would like it.

It is built from human preference data. People are shown a prompt with two candidate answers and pick the better one; the reward model is then trained so that the answer humans preferred receives a higher score than the one they rejected (the math for turning comparisons into scores is the Bradley-Terry model). Once trained, the reward model can score brand-new answers automatically, far faster and cheaper than asking a person each time, which is what makes the reinforcement-learning step practical: the policy model can be scored millions of times.

The crucial honest point is that a reward model is a learned proxy for human preference, not human preference itself. It is trained on a limited set of comparisons and will have blind spots; push a policy hard enough to maximise its score and the policy will discover answers that fool the reward model without genuinely being better, a process called reward over-optimization and an instance of Goodhart's law. This is why RLHF systems pair the reward model with a KL-divergence penalty and often refresh the reward model as the policy changes.