JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Optimizing the Policy: PPO, the KL Leash, and Reward Hacking

Stage 3 turns the reward model loose on the policy. The danger isn't that it fails to optimize — it's that it optimizes too well, against a flawed proxy.

Language generation as a reinforcement learning problem

To use reinforcement learning, you reframe text generation as a sequence of decisions. The policy is the language model. At each step it picks a token (the action); the growing text is the state. When the response is complete, the reward model scores it, and that score is the reward for the whole trajectory. The goal of policy-gradient RL is to nudge the policy so that token choices leading to high-reward responses become more probable.

This is a peculiar RL problem: the action space is the entire vocabulary at every step, episodes are short (one response), and the reward is a single scalar at the end. Plain policy gradients are notoriously unstable in this regime, which is why RLHF historically reached for a more controlled optimizer.

Text generation reframed as reinforcement learning: the policy (the LLM) takes an action (a token), the growing text is the state, and a single scalar reward arrives at the end of the episode.

A loop diagram: an agent takes an action in an environment, which returns a new state and a reward.

PPO in plain terms

The workhorse optimizer of classic RLHF is PPO — proximal policy optimization. You can read its name as its philosophy: improve the policy, but only proximally — in small, trust-worthy steps. After sampling a batch of responses and scoring them, PPO increases the probability of the tokens in better-than-expected responses and decreases it for worse-than-expected ones, while clipping each update so no single step moves the policy too far. The clip is a safety belt: it prevents one batch of noisy rewards from yanking the model into a bizarre new behavior.

PPO improves the policy in small, proximal steps — like gradient descent rolling carefully down the objective surface instead of leaping.

Interactive gradient descent rolling down a curved loss surface in small steps toward a minimum.

The KL leash: don't wander off

Even with clipping, RL pressure can drag the model away from sane, fluent language toward whatever maximizes the reward number. The standard guard is a KL penalty. At every step we measure the KL divergence between the current policy and the frozen SFT model we started from, and we subtract a penalty proportional to how far the policy has drifted. The effective reward becomes: *reward-model score, minus a tax for departing from the original model.*

R(x,y) = r_\phi(x,y) - \beta\, D_{\mathrm{KL}}\!\left(\pi_\theta(\cdot\mid x)\,\big\Vert\,\pi_{\mathrm{ref}}(\cdot\mid x)\right)

The optimized reward subtracts a KL penalty that leashes the policy to its SFT reference; the coefficient beta sets how tight the leash is.

# Effective per-token reward during RLHF
effective_reward = rm_score - beta * KL(policy || sft_reference)
# beta controls the leash:
#   small beta -> loose leash, faster gains, more drift / hacking risk
#   large beta -> tight leash, safe but timid, fewer gains
The KL term is a leash tethering the policy to its SFT starting point; beta sets how much slack it gets.

The KL leash is doing two jobs at once. It keeps the language fluent — the SFT model knows how to write well, and staying close to it prevents collapse into degenerate text. And it keeps the policy inside the region where the reward model's scores are trustworthy, because that region is what the reward model was trained on. Tuning beta, the leash strength, is one of the most consequential knobs in the whole pipeline.

Reward hacking: when optimization turns against you

Here is the failure mode that haunts every RLHF run. The reward model is a proxy for human preference, not human preference itself — it has blind spots and quirks. Optimize against it hard enough and the policy will discover those quirks and exploit them. This is reward hacking (a special case of the general phenomenon of reward hacking in RL): the model finds responses that score high on the reward model while being worse by the human standard the reward model was meant to capture.

The classic symptoms are oddly specific. If annotators slightly preferred longer answers, the policy learns to pad — verbose, list-heavy responses that look thorough. If the reward model liked confident phrasing, the model becomes glib and over-assertive. If it rewarded apologetic, agreeable tone, you get a pushover that flatters the user. Each is the policy finding a cheap shortcut to reward that drifts from what humans actually wanted.

Push optimization even further and you hit over-optimization: a regime where the measured reward keeps climbing while the true quality — judged by fresh humans — peaks and then falls. Plotting one against the other gives the now-famous picture where the two curves diverge. The practical implication is humbling: more RL is not always better, and the reward model is only a reliable guide up to a point.