The pain that DPO removes
Classic RLHF is powerful but fiddly. You maintain four models in memory at once (policy, reference, reward model, and a value head), you babysit the KL coefficient, and PPO is sensitive to a dozen hyperparameters that can quietly wreck a run. Many teams asked an obvious question: the reward model is just a stepping stone to a better policy — could we cut out the middleman and optimize the policy on the preferences directly?
Direct preference optimization (DPO) answers yes. Its key mathematical insight is that the RLHF objective — maximize reward subject to a KL leash — has a closed-form optimal policy, and you can rearrange the algebra so the reward model disappears entirely. What remains is a simple supervised-style loss defined directly on preference pairs. No separate reward model, no sampling loop, no PPO.
The KL-constrained RLHF objective has a closed-form optimum — and DPO inverts it to read the implied reward straight off the policy, with no separate reward model.
How DPO works, intuitively
The trick is that the language model can serve as its own implicit reward model. DPO defines an implied reward as the log-ratio between the current policy's probability of a response and the frozen reference model's probability of it. Then it applies the same pairwise ranking loss from guide 2 — push the chosen answer's implied reward above the rejected one's — but now "implied reward" is computed straight from the policy's own probabilities.
# DPO loss for one (chosen, rejected) pair # pi = policy being trained, ref = frozen reference delta_chosen = log pi(chosen) - log ref(chosen) delta_rejected = log pi(rejected) - log ref(rejected) loss = -log_sigmoid( beta * (delta_chosen - delta_rejected) ) # Raise prob of chosen, lower prob of rejected -- relative to ref.
Read the loss intuitively: it raises the probability the model assigns to preferred answers and lowers it for dispreferred ones, relative to the reference model. The same beta that controlled the KL leash in RLHF reappears here, controlling how aggressively the policy is allowed to move away from the reference. So DPO has not thrown away the KL idea — it has baked the leash directly into a single, stable loss.
The DPO loss: raise the preferred answer's implicit log-ratio reward over the dispreferred one, scaled by beta against the frozen reference.
A growing family of preference methods
DPO opened a flood of variants, collectively the direct preference optimization methods. Each tweaks an assumption DPO made. IPO addresses DPO's tendency to overfit when one answer is always preferred, by regularizing the objective so it can't push probabilities to extremes. KTO drops the need for pairs entirely — it learns from individual answers each labeled simply "good" or "bad," which is far cheaper to collect. ORPO folds preference learning into the SFT stage itself, doing instruction tuning and preference optimization in a single pass with no reference model at all.
A simpler cousin worth knowing is rejection sampling fine-tuning (sometimes called best-of-N): sample several responses per prompt, keep only the highest-scoring ones (by a reward model or a judge), and plain-SFT on those winners. It is crude but remarkably effective, and it is often used to bootstrap data before a DPO or RLHF pass.
DPO vs RLHF: which, and when
The honest answer to DPO versus RLHF is that both work, and the choice is about engineering constraints more than a clear winner. DPO is simpler, cheaper, more stable, and reproducible — the default for most fine-tuning today and ideal when you have a fixed preference dataset. Full RLHF with PPO remains attractive when you want on-policy learning: the model generates fresh responses, gets scored, and learns from its own current mistakes in a loop, which can reach quality DPO's static dataset cannot.