JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

The reward you cannot write down

Why some of the most important goals — be helpful, be honest, write a good summary — resist a hand-coded reward, and how human feedback steps in.

The goal you can describe but can't score

Standard reinforcement learning assumes someone hands you a reward function: a rule that turns each state and action into a number. For a game that is easy — the score is the reward. But suppose the goal is "summarize this article well" or "answer helpfully and honestly". You can recognize a good answer when you see one, yet you cannot write a clean formula that scores every possible answer. This gap is the reward specification problem, and it is exactly where ordinary RL stalls.

In standard RL the environment hands the agent a reward each step — but for goals like 'be helpful' there is no such number to give.

Diagram of the agent–environment loop: the agent takes an action, the environment returns a new state and a reward.

The trap is that any formula you do write is a proxy. Reward "summaries that are short" and the model learns to delete information. Reward "answers containing keywords" and it learns to stuff in keywords. The thing you can measure is never quite the thing you want.

Let people judge instead

Reinforcement learning from human feedback (RLHF) replaces the missing reward with human judgement. Instead of asking a person for an exact number, we put humans in the loop — this is human-in-the-loop RL — and let them say which behaviours they prefer. The broad family that learns policies from such judgements is preference-based RL.

Notice the shift in what we ask of a human. We are no longer demanding *"score this on a 1–10 scale"*. We are asking the far easier question *"which of these two is better?"*. That single design choice carries most of RLHF's reliability, and it deserves its own section.

Comparisons beat scores

There are two ways to gather human opinion. Scalar feedback asks a rater to assign an absolute number; comparative feedback asks them only to rank options. The distinction — scalar vs comparative feedback — matters because humans are noisy at absolute numbers and calibrated at comparisons. Your "7/10" drifts day to day; your "this one is better than that one" stays remarkably stable.

So RLHF is built on preference comparisons: show a rater two candidate behaviours and record which they prefer. A behaviour here can be a single answer, or for control problems a whole trajectory — a stretch of states and actions over time. Either way the raw material is the same: pairs, plus a thumb pointing at the winner.

P(A \succ B) = \sigma\!\left(r(A) - r(B)\right) = \frac{1}{1 + e^{-(r(A) - r(B))}}

The Bradley–Terry model: the chance a rater prefers A over B is a sigmoid of the two behaviours' reward gap.

The three moves of RLHF

Almost every RLHF system, from a robot to a chat model, is the same three moves in sequence. This is the RLHF recipe, and the rest of this track is one guide per move plus a guide on what breaks.

  1. Collect preferences. Gather many comparisons from people who know what good looks like (next guide).
  2. Fit a reward model. Train a model that predicts which behaviour a human would prefer, turning comparisons back into a usable number.
  3. Optimize the policy. Use ordinary RL to maximize that learned reward — while staying close to where you started so you don't exploit its blind spots.
The three moves of RLHF: collect human preferences, fit a reward model, then optimize the policy against it.

Pipeline diagram: human preference data feeds a reward model, which then trains the policy.

If you have met RL before, only the middle move is new: a reward that is itself a trained model rather than a fixed rule. If you haven't, don't worry — we build each move from intuition first.