The RLHF Pipeline: From Human Preferences to a Reward Model

The three stages, end to end

RLHF — reinforcement learning from human feedback — has a slightly misleading name, because most of the work happens before any reinforcement learning. The classic pipeline has three stages, and you should hold all three in your head at once because they hand off to each other.

1. Stage 1 — SFT. Fine-tune the base model on curated conversations so it behaves like an assistant at all. This is your starting policy.
2. Stage 2 — Reward model. Collect human preferences between pairs of answers, then train a separate model to predict which answer a human would prefer.
3. Stage 3 — RL optimization. Use the reward model as an automatic scorer, and reinforce the SFT model toward responses the reward model rates highly. (Guide 3 covers this stage.)

Notice that human effort concentrates in stage 2. People are slow and expensive, so we use them to label a finite set of comparisons, then distill their collective judgment into a reward model that can score millions of new answers automatically. The reward model is the bridge that lets a small amount of human feedback steer an enormous amount of training.

What preference data actually looks like

The raw fuel of RLHF is human preference data. The unit is not a graded essay but a comparison: take one prompt, sample two (or more) candidate answers from a model, and ask a person, "Which is better?" This pairwise comparison format is deliberate. Humans are unreliable at giving an absolute score — is this answer a 7 or an 8 out of 10? — but quite reliable at the relative judgment that answer A beats answer B.

{
  "prompt": "Explain why the sky is blue to a 10-year-old.",
  "response_a": "Sunlight is made of many colors. Blue light...",
  "response_b": "The sky is blue because of the atmosphere.",
  "chosen": "a",
  "annotator_notes": "A is clearer and age-appropriate; B is too terse."
}

Stack up tens or hundreds of thousands of these comparisons and you have a dataset that encodes, statistically, what your annotators collectively mean by a better answer — more accurate, clearer, safer, better formatted, more honest about uncertainty. The reward model's job is to learn that implicit standard.

Annotation: where quality is won or lost

Everything downstream inherits the quality of your preference data annotation. If annotators are vague, rushed, or disagree about what good means, the reward model learns a blurry, contradictory target — and the final assistant inherits that confusion. Strong annotation programs invest heavily in written guidelines that operationalize the abstract helpful-harmless-honest goals into concrete, checkable rules.

Annotation also has to cover the hard cases on purpose. Easy prompts where every answer is fine teach the reward model nothing. The valuable comparisons are the close calls and the edge cases — a borderline-unsafe request, a subtly wrong factual claim, an answer that is fluent but evasive. Curating prompts that probe exactly these boundaries is as important as the labeling itself.

Training the reward model

A reward model is usually the same transformer architecture as the policy, but with the language head replaced by a single scalar output: given a prompt and a response, it emits one number — a reward estimating how much a human would like that response. Training it uses the comparisons directly: for each pair, push the score of the chosen answer above the score of the rejected one. Formally this is the Bradley–Terry model of pairwise preference, but the intuition is simply "make winners outscore losers."

# Reward model loss for one comparison
#   r = reward_model;  chosen beats rejected
score_win  = r(prompt, chosen)
score_lose = r(prompt, rejected)
loss = -log_sigmoid(score_win - score_lose)
# Gradient pushes score_win up and score_lose down.

The pairwise ranking loss: the reward model is trained so the chosen answer outscores the rejected one.

One subtlety worth flagging now: the reward model only sees the distribution of answers it was trained on. Once stage-3 RL starts pushing the policy into new territory, it may generate responses unlike anything in the reward model's training set — and out there, the reward model's scores can be wildly miscalibrated. That mismatch is the seed of reward hacking, the central hazard of the next guide.