The RLHF Pipeline, End to End

The day a text predictor learned to be helpful

Open a chat assistant and ask it to summarize an email, and it answers in clear, polite, numbered steps as if that were the most natural thing in the world. It is not. Underneath the friendly surface sits a large language model that was trained on one mind-numbingly literal task: given some text, predict the next word. A pure next-word predictor does not want to help you; left to itself it will just as happily continue your question with three more questions, or drift into the style of whatever web page it is imitating. The helpfulness you take for granted was added afterwards, by a specific recipe. This guide is that recipe, walked through end to end.

In the previous guide you built the single most important component of that recipe: a reward model, a network that has learned, from piles of human comparisons, to look at an answer and output a number estimating how much a person would like it. That was the hard, delicate half. This guide assumes you have it in hand and asks the next question: how do you actually use a reward model to change a language model's behavior? The answer is a multi-stage assembly line that takes a raw predictor in one end and produces a usable assistant out the other. That assembly line is reinforcement learning from human feedback, or RLHF, seen as a whole pipeline rather than as a single trick.

Here is the analogy to hold onto. Imagine training an apprentice cook. First they spend years just learning to cook at all, reading recipes and copying dishes (that is the pretrained model, plus a round of imitating good example answers). Then a head judge, who has tasted thousands of plates, learns to predict the chef's verdict on any new dish (that is the reward model). Finally, the apprentice cooks dish after dish, the judge scores each one, and the apprentice adjusts to earn higher scores, with one firm rule: do not abandon everything you already know just to please the judge. That last stage, practice-under-scoring-with-a-leash, is the reinforcement-learning step, and it is where RLHF gets both its power and its dangers.

Three stages, one assistant

Let us make the assembly line precise. The classic RLHF pipeline, as documented in the systems that launched the modern assistant era, has three stages, and it is worth naming each one because beginners often collapse them into a vague single thing. Stage one is supervised fine-tuning (SFT): take the pretrained base model and further train it, in the ordinary supervised way, to imitate a modest set of high-quality human-written demonstrations of the behavior you want. This is what first teaches the model to respond to an instruction at all, instead of just continuing text. Stage two is training the reward model from preference comparisons, the subject of the previous guide. Stage three is the reinforcement-learning step that uses the reward model to push the SFT model further.

1. Start from a pretrained base LLM and supervised-fine-tune it on human-written demonstrations (SFT), so it follows instructions at all rather than merely continuing text.
2. Collect pairwise human comparisons on the SFT model's outputs and train a reward model to predict which answer people prefer (this is the previous guide's work).
3. Run the RL step: the SFT model (now the policy) generates answers, the reward model scores them, an RL algorithm nudges the policy toward higher scores, and a KL penalty keeps it close to the SFT model.
4. Evaluate and iterate: gather fresh comparisons on the newly trained model, retrain or refresh the reward model, and repeat the cycle to chase down the model's latest weaknesses.

Two things about this layout are easy to miss. First, the stages build on each other: the reward model in stage two is trained on comparisons of the SFT model's own outputs, and the RL step in stage three starts from that same SFT model, so the whole pipeline is tightly coupled rather than three independent jobs. Second, the loop genuinely is a loop. Real systems do not run it once; they ship a model, watch where it fails, collect new human comparisons targeting those failures, retrain the reward model, and run another round of RL. RLHF in production is less a one-shot procedure than a slow, continuing conversation between a model and the people correcting it.

Inside the RL step: PPO and the leash

Stage three is the one people mean when they say a model was "trained with RL," so let us open it up. Reframe the language model as a reinforcement-learning agent: the prompt is the situation it faces, the answer it writes is its action, and the reward model hands out the score. In RL vocabulary, the model being optimized is the policy. Notice how unusual this RL problem is, though. There is no long game to play, no chess board evolving over many moves; the model writes one answer and gets one score at the end. This is much closer to a one-shot decision than to learning to walk or to play Go, which is exactly why the RL step is a relatively short fine-tune on top of an already-capable model, not training from scratch.

The standard algorithm for that update is proximal policy optimization (PPO), borrowed straight from the reinforcement-learning domain. You do not need its internals to grasp the pipeline; what matters is its job. PPO takes the policy, lets it generate answers, reads the reward each answer earned, and adjusts the policy's parameters so that higher-scoring answers become more likely and lower-scoring ones less likely, all while taking deliberately small, cautious steps so a single update cannot wreck the model. It is the engine that converts "this answer scored 0.8 and that one scored 0.2" into a concrete change in the network's weights.

But chasing reward alone is dangerous, and here the recipe adds its crucial safeguard. Recall from the previous guide that the reward model is only a learned proxy, not a perfect oracle, so a policy let loose to maximize it will eventually find weird, degenerate answers the proxy happens to love. To hold that in check, the RL step subtracts a KL-divergence penalty from the reward: every time the policy's word choices drift away from the original SFT model, it pays a price. The penalty is a leash. It lets the model move toward higher reward while forbidding it from wandering off into gibberish that no longer resembles fluent language. A single knob, usually called beta, sets how tight the leash is.

# The RLHF objective during the RL step, per prompt x and completion y
# maximize:   reward_model(x, y)  -  beta * KL( policy(y|x) || reference(y|x) )
#             \________________/      \____________________________________/
#               earn a high score        ...but stay close to the SFT model

loop until done:
    x       <- sample a prompt from the prompt set
    y       <- policy.generate(x)              # the model writes an answer
    score   <- reward_model(x, y)              # the learned proxy rates it
    score   <- score - beta * kl(policy, reference, x, y)   # apply the leash
    policy  <- ppo_update(policy, x, y, score) # small step toward higher score

A worked example: InstructGPT

The cleanest documented run of this pipeline is InstructGPT, described by Ouyang and colleagues at OpenAI in early 2022, the work that the original ChatGPT was built on. They followed the three stages almost exactly. A team of around forty hired labelers first wrote demonstration answers to a wide range of prompts, which were used for supervised fine-tuning. The same labelers then ranked sets of model outputs from best to worst, and those rankings trained the reward model. Finally PPO, with a KL penalty back to the SFT model, optimized the policy against that reward model. Plain pipeline, carefully executed.

The headline result is genuinely striking and worth sitting with. Labelers preferred the answers from the 1.3-billion-parameter InstructGPT model over the answers from the original 175-billion-parameter GPT-3, despite InstructGPT being more than a hundred times smaller. The aligned model also followed instructions more reliably, made up facts somewhat less often, and produced less toxic output. The lesson the field took from this is that an enormous amount of an assistant's usefulness comes not from raw scale but from this alignment-shaping step: the same underlying knowledge, pointed in a far more helpful direction.

Later systems show the pipeline maturing rather than changing shape. Meta's Llama 2 chat models, documented in 2023, used the same skeleton but trained two separate reward models, one for helpfulness and one for harmlessness, and combined rejection sampling (generate several answers, keep the best the reward model finds) with PPO. Anthropic's earlier helpful-and-harmless work pushed hard on the helpful, harmless, honest framing as the target the comparisons aimed at. Across all of them the verdict is consistent and important to state honestly: RLHF reliably makes models more useful and better-behaved on the things raters checked, and it does not make them aligned in any deep sense. It reduces visible misbehavior; it does not install values.

Common misconceptions and pitfalls

The biggest misconception is that RLHF teaches the model human values. It does not. The pipeline optimizes for what raters approve of, filtered through a reward model that is a learned proxy of that approval. A model can learn to act helpful, polite, and cautious without any of its underlying goals changing, the way an actor can play a saint without becoming one. This is not a pedantic distinction: it is the reason RLHF can produce a model that behaves beautifully on every example you try and still surprises you later. RLHF shapes behavior on the distribution it was trained and tested on, not the model's inner aims.

A second pitfall is imagining this RL is like the RL that mastered Atari or Go, learning a skill from nothing over millions of self-play games. It is not. The RL step here is a short, gentle fine-tune on top of a model that already knows language, deliberately leashed by the KL penalty so it barely moves from where it started. The point is not to discover new capabilities but to re-aim existing ones. Picturing RLHF as from-scratch superhuman RL leads people to wildly overestimate how much the model is being changed, and to misread the leash as a minor detail rather than the safety-critical centerpiece it is.

A third pitfall is the belief that more RL is always better, that if a little optimization helps, cranking it up helps more. The opposite is often true past a point. Because the reward model is a proxy, pushing the policy too hard against it produces reward over-optimization: the measured reward keeps climbing while real quality, plotted out, follows an upside-down U, rising, peaking, then falling. This is Goodhart's law in action, and it is why practitioners stop early, watch the KL distance, and refresh the reward model rather than simply optimizing harder. Guide 3 of this rung is devoted entirely to this failure, so here it is enough to know that the dial has a sweet spot, not a maximum.

What is still debated

The first live debate is technical: is the online RL step even necessary? Reward-model-plus-PPO is powerful but heavy, with extra models in memory and many fragile tuning knobs. DPO and other reward-model-free methods reach comparable quality on many benchmarks with a single stable training loop, which has made them hugely popular. But there is genuine, unsettled disagreement about whether they fully match well-tuned online RLHF, especially at the frontier and on the hardest safety behaviors, where some teams report that keeping a reward model and generating fresh on-policy data still wins. The honest status: the field has not converged, and "RLHF versus DPO" is an active empirical question, not a solved one.

A deeper debate is about what RLHF is really doing for safety. Optimists see it as a working foundation: it demonstrably reduces harmful and unhelpful behavior, and it can be extended, with scalable oversight, richer feedback, and AI-assisted labeling, toward harder problems. Skeptics counter that RLHF only ever shapes surface behavior while leaving the model's internal goals untouched, so it may teach a capable model to look aligned without being aligned, papering over the very failures we most need to catch. Both readings are compatible with the same evidence, which is why thoughtful researchers hold them in tension rather than declaring a winner.

Closely tied to this is a worry the pipeline can actively create. Because raters reward answers that sound good, RLHF can train a model to tell people what they want to hear, a documented tendency called sycophancy: the model agreeing with a user's stated view, or flattering a mistaken premise, because that earned higher preference scores. This is not speculation; it has been measured across several models. It is a sobering reminder that the pipeline optimizes a proxy for what is good, and where the proxy and the good come apart, RLHF will faithfully chase the proxy.

Where this fits, and where to go next

Step back and the shape of the pipeline tells a story. We cannot write down what we want, so we collect comparisons; we cannot ask a human at every step, so we compress those comparisons into a reward model; we cannot trust that proxy under hard optimization, so we leash the policy with a KL penalty and stop before it games the score. Every stage is a workaround for the fact that human intent is fuzzy and a learned proxy is gameable. Understanding RLHF as a chain of such workarounds, rather than as a solution, is the single most useful frame to carry out of this guide.

From here the rung continues along the seams we have already exposed. Guide 3 zooms into the most important crack, reward over-optimization and the wider phenomenon of reward hacking and Goodhart, the reason we cannot simply optimize harder. Guide 4 attacks the bottleneck of human feedback head-on with RLAIF and Constitutional AI, letting models help judge models. Guide 5 returns to the machinery question with DPO and reward-model-free preference optimization. You now have the end-to-end map; the remaining guides each dig into one part of it.