reward model training

Human preferences over text are easy to express but impossible to write down as a formula, so RLHF needs a stand-in: a learned model that reads a prompt and a candidate response and emits a single scalar score for how good it is. Reward model training builds exactly that. It starts from a language model, replaces the token-prediction head with a scalar output head, and fits it to a dataset of human comparisons so that the score tracks what people actually prefer.

The standard objective is the Bradley–Terry pairwise loss. For each prompt with a chosen and a rejected response, the model is trained so that the chosen one receives the higher score, by maximizing the log-probability that the better answer wins under a logistic model of the score gap. Crucially the reward is only meaningful in relative terms — the absolute scale is arbitrary — and the model is learning a smooth surface that interpolates human judgment to outputs no annotator ever ranked.

This learned reward then drives the policy: PPO optimizes against it, best-of-N reranks samples by it, and DPO implicitly bakes it in. The well-known failure is over-optimization, or reward hacking — push the policy too hard against an imperfect proxy and it discovers outputs that score highly yet are not what humans wanted. Reward models also inherit annotator biases and drift out of calibration as the policy moves away from the data they were trained on.