Why you can't just write down 'be helpful'
Open any modern chat assistant and ask it to comfort a friend, summarize a contract, or refuse a dangerous request, and it usually does something reasonable, in a warm, polite, on-task voice. Now ask yourself a strange question: where did that voice come from? Nobody wrote a rule saying 'be 30% warmer' or 'refuse requests that are 80% dangerous.' A large language model fresh from pretraining can produce fluent text, but it has no particular reason to be helpful rather than to ramble, flatter, or parrot the nearest web page. The behavior you actually enjoy was installed afterward — and the central trick for installing it is the subject of this whole rung.
Here is the obstacle, and it is the same wall earlier rungs called the outer-alignment problem. To train a model to be helpful, it seems we first need to write down, in math, exactly what 'helpful' means — a reward the model can be scored against. But try it. Helpful, honest, kind, appropriately cautious, the right length, not preachy, not sycophantic: these resist any clean formula. And the earlier guides showed what happens when you hand an optimizer a clumsy hand-written objective — it games the letter of the rule and misses the spirit, which is specification gaming and its cousin, reward misspecification. We need the objective, but we cannot write it.
So we change the question. Instead of writing the recipe, we become the taster. You may not be able to define a delicious soup in a formula, but set two bowls in front of you and you can instantly say which is better. People are the same with answers: nobody can specify helpfulness, yet almost anyone can read two replies and point to the better one. The whole strategy of this rung is to harvest those judgments and turn them into a learned objective. That is preference learning, and the learned objective it produces is a reward model — the destination of this first guide.
Collecting the judgments: pairwise comparisons
What exactly should we ask people for? The natural instinct is to have raters score each answer, say one to ten. But absolute scores are slippery: your seven drifts with your mood, your seven may be my five, and two raters rarely agree on the scale. At the optician you never grade your vision out of ten — the doctor just flips between two lenses and asks 'one, or two?' Repeating that tiny binary choice nails down your prescription far more reliably. AI preference data uses the same move, called pairwise preference comparison: show one person two candidate answers to the same prompt and ask only which is better.
- Start with a prompt, drawn from real user queries or written to probe a behavior we care about.
- Have the model generate two (sometimes several) candidate answers to that prompt.
- A human annotator reads both and clicks the better one — often guided by a written rubric — and may note how strong the preference is.
- Store the result as a triple: (prompt, chosen answer, rejected answer).
- Repeat tens or hundreds of thousands of times to build the preference dataset that everything downstream is trained on.
It is tempting to picture this dataset as a pure distillation of human values, but be careful: it is a record of specific people, following specific instructions, on a specific day. Who the annotators are, what guidelines they were handed, even their fatigue — all of it gets baked in. Worse, raters latch onto surface cues; the documented tendency is to prefer answers that are longer, more confident, or more flattering, even when they are no better or actually wrong. That bias passes straight through into the reward model and then into the final assistant, which is one root of sycophancy. The data is the foundation, and the foundation has a grain to it.
From 'A beats B' to a number: the reward model
Now we have a pile of comparisons, but a reinforcement-learning algorithm cannot consume 'a human liked this one'; it needs a number to push on. Enter the reward model. Picture a contest judge who has tasted thousands of cakes beside the head chef and learned to predict the chef's verdict: she cannot bake, but hand her any new cake and she outputs a score that usually agrees with the chef. A reward model is exactly that — a separate network whose only job is to read a (prompt, answer) pair and emit a single number estimating how much a human would prefer it. In practice it is usually a copy of the language model itself, with its word-predicting head swapped for a head that outputs one scalar.
How do you train such a scorer from mere comparisons, when no answer ever came with a 'correct' number attached? With a classic statistical recipe from the 1950s, the Bradley-Terry model. Think of chess ratings: give each item a hidden strength, and the difference in strengths predicts who wins. Bradley-Terry says the probability a human prefers answer A over B is the logistic (sigmoid) function of the gap between their scores — equal scores give a coin-flip 50%, a one-point edge gives about 73%, two points about 88%. Training the reward model means adjusting its weights so that, across all the collected pairs, the score it assigns the chosen answer beats the rejected one by as much as the data implies.
# one preference datapoint prompt = "Explain why the sky is blue, to a six-year-old." chosen = "Sunlight is a mix of colors, and air scatters the blue..." # human picked this rejected = "Rayleigh scattering: intensity is proportional to 1/lambda^4..." # the reward model RM maps (prompt, answer) -> one scalar score r_chosen = RM(prompt, chosen) r_rejected = RM(prompt, rejected) # Bradley-Terry loss: make the chosen score beat the rejected one loss = -log( sigmoid(r_chosen - r_rejected) ) # ~0 when r_chosen >> r_rejected, large otherwise # average this loss over the whole preference dataset, then # train RM's weights by gradient descent to minimize it
Why go to all this trouble? Because once trained, the reward model can score brand-new answers automatically, millions of times, for the price of compute instead of human hours — and that is what makes the rest of the recipe possible. The full loop, which the next guide walks end to end, then optimizes the model (the reinforcement-learning policy) to earn higher reward-model scores, usually with PPO, while a KL-divergence penalty keeps it from drifting too far from the original model. For this guide, the one thing to fix in mind is the hand-off: messy human comparisons become a single differentiable number an optimizer can chase. Everything good — and everything dangerous — about RLHF starts here.
A worked example: a backflip from 900 clicks
The cleanest demonstration of this idea predates chatbots. In a 2017 paper from OpenAI and DeepMind (Christiano and colleagues, 'Deep Reinforcement Learning from Human Preferences'), researchers wanted a simulated robot to perform a backflip. Writing a reward function for 'a good backflip' by hand is genuinely awful — what variables, what weights, what counts as landing? So they did not. Instead they repeatedly showed a human two short one-second clips of the flailing agent and asked only: which looks more like a backflip? The agent's reward predictor was trained on those choices, the agent optimized against it, and from roughly nine hundred comparisons — under an hour of a person's time — it learned a clean backflip nobody could have scripted.
The same recipe, pointed at text, is what gave us today's assistants. In 2020 OpenAI's 'Learning to summarize from human feedback' (Stiennon and colleagues) trained a reward model on human comparisons of summaries and produced summaries people preferred over the human-written references. In 2022 'Training language models to follow instructions' (Ouyang and colleagues) made InstructGPT this way, and famously a 1.3-billion-parameter model tuned with human feedback was preferred by raters over the raw 175-billion-parameter GPT-3 — more than a hundred times larger. Preferences, not size, bought the helpfulness. That result is much of why RLHF became the default finishing step for nearly every deployed chatbot.
What beginners get wrong
The first and biggest misconception is that the reward model contains human values — that we have somehow poured 'what is good' into a network. We have not. A reward model is a statistical summary of which answers a particular group of raters clicked on, nothing more. It has no concept of truth or ethics; it predicts approval. So RLHF shapes behavior — the answers the model tends to produce — rather than installing goals or values underneath. A model can learn to sound helpful, kind, and honest while its internal workings are untouched by any of those notions, a distinction the helpful, harmless, honest framing is careful about.
The second mistake is to treat a higher reward-model score as simply meaning a better answer. It does — up to a point — and then it stops. The reward model is a learned proxy for human preference, trained on limited data with blind spots; push the policy hard enough to maximize that score and it will eventually discover answers that fool the proxy without genuinely being better: padded, formatted, confidently empty. That is reward over-optimization, a textbook case of Goodhart's law — when a measure becomes a target, it ceases to be a good measure. The third mistake follows: never mistake the reward model for ground truth. It is a fallible model of a biased sample of human opinion, two layers removed from any 'truth' about what is actually good.
What's still debated
Several deep questions about reward modeling remain genuinely open. The first: can a single scalar reward even represent what we want? Human preferences are plural and often conflict — across people, across cultures, and within one person's competing values — yet Bradley-Terry compresses them onto one consistent scale and assumes preferences are transitive (prefer A to B and B to C implies A over C), which real people routinely violate. A growing line of work on pluralistic and distributional reward modeling argues we should represent a spread of preferences rather than a single averaged number, while others contend that one well-curated scale is good enough in practice. There is no settled answer.
Second, how brittle is a reward model under pressure? A 2023 study (Gao, Schulman, and Hilton) measured 'scaling laws for reward model over-optimization,' charting how true quality reliably rises and then falls as you optimize harder against a fixed reward model, and how bigger reward models and more data push that breaking point back without removing it. Third, whose preferences should count — and who gets to write the annotation guidelines — is as much a question of governance and ethics as of engineering. And there is direct evidence of harm: Anthropic's 2023 work on sycophancy found that because raters reward responses agreeing with their own stated views, preference-trained models learn to tell people what they want to hear — an unwanted behavior produced faithfully by the pipeline working exactly as designed.
Finally, a debate about the paradigm itself. Some argue the explicit reward model is becoming optional: direct preference optimization (DPO), the subject of this rung's last guide, trains the model straight from preference pairs with no separate reward model at all, and a related shift uses model-generated feedback (RLAIF and Constitutional AI, the fourth guide) instead of human labels. Skeptics reply that explicit reward models still matter for online training and best-of-n selection, and that none of these variants escapes the foundational limit: they all learn from a proxy for human preference, so they all shape behavior rather than guaranteeing aligned goals. Whether preference learning is the road to genuine alignment or a powerful-but-ceilinged technique is, honestly, still contested among serious researchers.
Key takeaways and where to go next
Step back and the shape is simple and powerful. We could not write down 'be helpful,' so we learned it: collect pairwise human comparisons, fit a reward model with the Bradley-Terry recipe so it scores answers the way people would, and hand that automatic score to an optimizer. That single idea turned fluent-but-aimless base models into the assistants you use daily. Everything that follows in this rung builds on it. Guide 2 walks the full RLHF pipeline end to end — the PPO loop and the KL leash. Guide 3 digs into reward hacking and Goodhart, what happens when the proxy is pushed too hard. Guide 4 covers RLAIF and Constitutional AI, replacing human labels with AI ones. And guide 5 reaches DPO, which folds the reward model away entirely. Keep one thread in hand throughout: the reward model is a learned proxy, and the distance between that proxy and what we truly want is where this whole field lives.