The three-part machine, and a shortcut
Picture a small research team that has just spent weeks collecting a clean pile of preference pairs: thousands of prompts, each with a better answer and a worse one. By the earlier guides in this rung they know exactly what to do with them. Train a reward model on the pairs, then run a reinforcement-learning loop, usually PPO, that lets the model generate answers, scores them, and nudges the weights upward, all while a KL penalty keeps it from drifting. It works. It also means holding several large models in memory at once, babysitting a famously twitchy RL loop, and tuning a thicket of knobs. The team's honest question is the one this guide answers: is all of that machinery actually necessary?
Here is the shortcut in plain language. The reward model in classic RLHF is really a middleman: it reads your preference pairs, compresses them into a score, and that score is what the RL loop chases. But the thing you ultimately wanted to change was the language model itself. So why not cut out the middleman and teach the model directly from the same pairs, raising its probability of producing the answers people preferred and lowering its probability of the ones they rejected? That single move, train the policy straight on the comparisons, no separate reward model, no RL loop, is the heart of direct preference optimization (DPO), the method that reshaped how open models get aligned.
Keep one thread from the earlier guides firmly in hand as we go, because it will not disappear no matter how we rearrange the machinery. Everything in this rung learns from the same raw material, pairwise comparisons, and from the same simple statistical story about how a strength gap turns into a win probability. DPO and its successors change how that signal is turned into a trained model; they do not change what the signal is, or its limits. So the lessons you already met, that a learned preference is a proxy and proxies can be gamed, will follow us right into the simpler methods.
What DPO actually does: a reward model in disguise
Now the actual mechanism, introduced one step at a time. RLHF is not chasing reward blindly; it is solving a specific problem, get the highest reward you can while staying close to the original base model (that closeness is enforced by the KL penalty you met earlier). It turns out this problem has a known, exact answer: the ideal RLHF policy assigns each answer a probability proportional to the base model's probability times the exponential of its reward. That is a tidy mathematical relationship between three things, the trained policy, the base model, and the reward.
DPO's clever move is to read that relationship backwards. If the policy is fixed by the reward, then the reward is also fixed by the policy: you can write each answer's reward as how much more (or less) likely the trained model is to produce it than the base model, scaled by a temperature. The title of the original 2023 paper by Rafailov and colleagues puts it memorably: your language model is secretly a reward model. Substitute that expression into the standard Bradley-Terry formula for the probability that the chosen answer beats the rejected one, and a wonderful cancellation happens, the awkward normalizing term drops out, and you are left with a plain loss you can minimize with ordinary supervised-style training.
# Classic RLHF: three moving parts, an RL loop, models held in memory
reward_model = train_RM(preference_pairs) # stage 1
policy = PPO(base_model, reward_model, beta) # stage 2: RL loop + KL penalty
# DPO: one model, one supervised-style loop, no reward model, no RL
# implicit reward of answer y to prompt x:
# r(x, y) = beta * log( policy(y|x) / base_model(y|x) )
loss = -log sigmoid( beta * ( log policy(y_win |x) / base_model(y_win |x)
- log policy(y_lose|x) / base_model(y_lose|x) ) )
# one gradient step pushes P(y_win) up and P(y_lose) down;
# base_model stays frozen as the reference -> an implicit KL leash, baked inTwo features of that loss are worth pausing on. First, the base model has not vanished; it appears as a frozen reference model in every term, which means the original KL leash is still present, just folded silently into the objective rather than added as a separate penalty. DPO did not abandon the constraint that kept RLHF stable; it absorbed it. Second, there is no longer a reward model sitting in memory to be optimized against, and no sampling-and-scoring RL loop, so the whole thing trains about as simply and stably as fine-tuning a large language model on labelled data. One model, one loop, far fewer knobs.
DPO in the wild: the open-model wave
The cleanest demonstration of why this mattered is a small model called Zephyr-7B, released by the Hugging Face H4 team in late 2023. They took an open base model (Mistral-7B), did a round of ordinary instruction tuning, and then aligned it with DPO on a public preference dataset, no reward model, no PPO, no proprietary human-labelling pipeline. The recipe was so light it could be reproduced on modest hardware, and yet on chat benchmarks the 7-billion-parameter result held its own against much larger models that had been aligned with the full, expensive RLHF machinery. The preferences in that dataset, incidentally, were generated by AI judges, exactly the RLAIF idea from the previous guide, showing how naturally these pieces snap together.
Zephyr was not a one-off. Because DPO is reproducible, cheap, and stable, it spread fast through the open ecosystem, AI2's Tulu models, university and community fine-tunes, and a flood of preference-tuned releases on model hubs all leaned on it. It also reached the frontier: Meta's published account of its Llama 3 models describes a post-training pipeline that uses DPO for preference alignment rather than a PPO-style RL loop. The practical effect was a quiet democratization, aligning a capable assistant stopped requiring a large RL team and became something a graduate student or a small lab could actually do.
Beyond DPO: an alphabet of preference methods
DPO opened a floodgate. Once researchers saw that you could fold preference learning into a single direct loss, a whole family of variations followed, each tweaking what to optimize, what to regularize toward, or what kind of feedback to accept. You do not need to memorize the acronyms, but it helps to see the shape of the space, because each one is a deliberate answer to a specific weakness, and together they show that DPO was a starting point, not a final word.
- IPO trims an overfitting habit. Researchers noticed DPO can push too hard when a preference looks near-certain, driving the model to extremes; IPO (identity preference optimization) reshapes the objective so it cannot run away like that.
- KTO drops the need for pairs. Instead of 'A is better than B', KTO (named after the psychologists Kahneman and Tversky) learns from a single thumbs-up or thumbs-down on one answer, which is exactly the kind of cheap, abundant signal real products collect.
- ORPO and SimPO try to drop the reference model. Both aim to remove the frozen base model from the loss, ORPO by folding instruction tuning and preference into one stage, SimPO by scoring answers with a length-normalized average instead, cutting memory and steps further.
- Online and iterative DPO close the freshness gap. Plain DPO learns from a fixed dataset; the iterative version repeatedly generates new answers, has a judge model (RLAIF again) label them, and retrains, recovering some of the on-the-fly feedback that PPO gets for free.
- DPO-Positive patches a documented quirk. Plain DPO sometimes lowers the probability of the preferred answer too (it only has to lower the rejected one more); DPOP adds a term that keeps the chosen answer's probability from collapsing.
Step back and the family resemblance is obvious. Every one of these is still preference learning dressed in a slightly different loss, still resting on the same comparison data and the same Bradley-Terry intuition, still applying some form of leash so the model does not wander off. It is worth contrasting them with one method that sidesteps training entirely: best-of-n sampling, which leaves the weights untouched and simply generates several answers at use-time and keeps the best. DPO bakes the preference into the model once; best-of-n pays at every query. Knowing both ends of that trade is part of reading the field fluently.
Simpler is not safer: misconceptions and pitfalls
The most tempting misconception is that by deleting the reward model, DPO escapes the proxy problem that haunted RLHF. It does not. The reward did not disappear; it became implicit, hidden inside the policy's own probabilities, as the 'secretly a reward model' framing makes plain. An implicit proxy is still a proxy, and it can still be over-optimized. In fact, a 2024 study by Rafailov and colleagues measured reward over-optimization in these direct methods and found the familiar hump-shaped curve, quality rising then falling as you push harder, the same Goodhart dynamic, just wearing different clothes. Simplicity bought convenience, not immunity.
A second misconception is that DPO 'got rid of reinforcement learning' in some deep sense. It removed the explicit RL loop, which is a real engineering win, but it also quietly changed something. PPO is on-policy: it scores the model's own fresh answers as it learns, so it always gets feedback about where it currently is. Plain DPO is offline: it learns from a fixed pile of pairs that some other model generated, and never sees the new regions its updates push it into. That gap is exactly why the iterative and online variants exist, and it is one honest reason DPO and online reinforcement learning can behave differently in practice rather than being mere notational rewrites of each other.
The third mistake is the same one that shadows all of RLHF, and it survives the switch to DPO untouched: believing that a model trained this way has acquired good values. It has acquired behaviour, the tendency to produce answers like the preferred ones in your dataset. Every bias baked into those comparisons rides along: a taste for longer or more confident replies, the cultural slant of whoever labelled them, and the pull toward sycophancy, telling the user what they want to hear. DPO learns these just as faithfully as RLHF does, because it is trained on the very same kind of data. Cheaper alignment is not more genuine alignment.
What's still debated: DPO versus PPO, and the alphabet soup
The liveliest open question is blunt: is the simpler method actually as good? Through 2024 this became a real back-and-forth in the literature. One influential study argued that a properly tuned PPO-style RLHF pipeline can outperform DPO, especially on harder reasoning and code tasks and on inputs unlike the training pairs, suggesting the offline shortcut leaves quality on the table. Defenders reply that DPO reaches comparable quality at a fraction of the cost and engineering risk, that many reported PPO wins depend on heavy tuning few teams can afford, and that the iterative and online variants close much of the gap. The honest status is unsettled: which method wins depends on the task, the data, the compute budget, and how carefully each was tuned.
Underneath the benchmark scuffle sits a deeper, genuinely unresolved question: is learning purely from a fixed preference set fundamentally limited? Optimists point out that a separate reward model can be reused, ensembled, and probed for weaknesses in ways an implicit reward cannot, and that on-policy exploration may matter for the hardest behaviours, hints that the reward model was doing useful work we should not be too quick to discard. Skeptics counter that these are engineering frictions, not laws, and that direct methods plus a judge model to refresh the data recover most of the benefit. Notice that none of this is settled by a single experiment; like the scaling-law extrapolations you met earlier, it is an empirical pattern still being mapped.
Finally, there is healthy disagreement about the alphabet soup itself. Is the steady stream of new *PO methods real conceptual progress, or partly a race up crowded leaderboards where small, fragile gains get over-celebrated? Both can be true at once. The cautious reading, and a good habit for a learner, is to treat each new method as a hypothesis that solved a specific weakness on specific benchmarks, hold the headline numbers loosely, and watch for independent replication before believing any one method is the new default. What is not in dispute is the bigger picture: these methods made preference alignment dramatically more accessible, while leaving the field's central proxy problem exactly where they found it.
Where this leaves the rung
Step back over the whole rung and a single arc comes into focus. We began by turning fuzzy human wishes into pairwise comparisons and then into a reward model; we ran that reward through the full RLHF and PPO pipeline; we watched it get gamed through reward hacking, over-optimization, and Goodhart; we scaled the feedback itself with RLAIF and Constitutional AI; and now we have folded the whole apparatus down to a single direct loss. The throughline is that all of it is preference learning, and all of it inherits both the power and the proxy-shaped limits of learning from what humans (or AI judges) prefer.
That limit is also a signpost. If every method in this rung shapes behaviour rather than reaching the goals underneath, then the natural next question is whether we can look inside a trained model and actually read what it learned, rather than only judging it from the outside. That is the work of interpretability, the focus of the next rung, where the field tries to open the black box and check whether the alignment we trained for is really there, or only appears to be. You leave this rung knowing how today's assistants are aligned, why the signal that aligns them can be gamed, and why the simplest method on the menu is a real gift to practitioners without being a solution to the deeper problem.