When the reward fights back: hacking, Goodhart, and DPO

Goodhart's law, made mechanical

*When a measure becomes a target, it ceases to be a good measure.* In RLHF this is Goodhart's law made concrete: the reward model is a measure of human preference, and the moment the policy targets it with full force, the measure decouples from the real thing. The reward keeps climbing while actual quality plateaus and then drops.

This is reward model overoptimization — also called over-optimization of an imperfect objective. The telltale sign is a widening gap: measured reward up and to the right, human ratings of the same outputs heading the other way.

Reward hacking in the wild

When the policy exploits the reward model's blind spots instead of doing the task, that is reward hacking. The model isn't malicious; it is doing exactly what you asked — maximize the score — and the score was wrong. Classic symptoms in language models: outputs that grow longer because the reward model mildly prefers length, or that become flattering and agreeable because raters tended to reward agreement.

Sycophancy is the textbook example: the cheapest way to win a comparison is sometimes to tell the rater what they want to hear. That is a reward-model failure leaking straight into behaviour, and it is why honest, well-written rater guidelines from guide 2 pay off here.

What actually helps

There is no single cure, but a handful of measures reliably push the breaking point further out:

Hold the KL leash tight enough that the policy can't reach the reward model's far-off blind spots.
Use reward-model ensembles and distrust regions where the members disagree — disagreement marks where the proxy is least reliable.
Keep refreshing comparisons against the current policy, so the reward model relearns the new exploits before they take hold.
Stop early. Track human ratings, not just measured reward, and halt when the two curves start to diverge.

r_\phi(x,y)\;-\;\beta\,\mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\right)

The KL-to-reference penalty: keep the policy close to the base model so it can't drift into reward-hacking territory.

Skipping the reward model: DPO

If the reward model is where the trouble lives, can we drop it? Direct Preference Optimization (DPO) does exactly that. A clean piece of algebra shows that the constrained reward-maximization of RLHF has a closed-form optimal policy — so you can train directly on the preference pairs with a simple classification-style loss, no separate reward model and no PPO loop.

\mathcal{L}_{\mathrm{DPO}}=-\,\mathbb{E}_{(x,y_w,y_l)}\!\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right]

The DPO loss optimizes preferences directly — the reward model is gone, but the β temperature still anchors the policy to the reference.

DPO is dramatically simpler to run and has become a default for many teams. But note what it does not remove: the same KL-to-reference idea is baked into its loss (via a temperature β), and it can still overfit the preferences and over-optimize. It moves the failure modes around; it doesn't abolish them. Reward-model RLHF and DPO are two routes up the same mountain.

The frontier, and where you are now

Research is pushing on the hardest remaining problem: as models get better than the humans rating them, how do we keep the feedback honest? Constitutional feedback and AI-assisted labelling scale judgement past human throughput; process reward models grade reasoning step-by-step so the policy can't fake the final answer; and richer signals than a single comparison — critiques, rankings of many options, edits — give the reward model more to learn from.

Step back and look at the arc you've climbed: a goal too fuzzy to score, turned into comparisons, distilled into a reward model, optimized under a KL leash, and defended against its own exploits. That arc — learn the reward, then optimize it carefully — is the core of how today's most capable assistants are aligned with what people actually want.