JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Alignment Without the RL Machinery: Preference Optimization

RLHF works but is heavy and finicky. A wave of methods — DPO, IPO, KTO, ORPO — aligns models from preference data with a plain supervised loss. Know what each fixes and when to use it.

Recap: why RLHF is heavy

Classic RLHF aligns a model in three stages: SFT, then train a reward model on human preference pairs, then optimize the policy against that reward with PPO. It works — it produced the first genuinely helpful assistants — but the RL stage is operationally painful. You juggle four models in memory (policy, reference, reward, value), tune a brittle KL penalty, and watch for reward hacking where the policy games the reward model instead of getting better.

RLHF's three-stage pipeline — SFT, then a reward model, then PPO — is exactly the heavy machinery preference optimization sets out to replace.

Diagram of the RLHF pipeline: human preferences train a reward model that then tunes the policy.

The insight that unlocked everything was Direct Preference Optimization (DPO): you can skip the explicit reward model and the RL loop entirely. DPO shows that the RLHF objective has a closed-form optimum, and you can rearrange it into a simple supervised loss directly on preference pairs. One model, one frozen reference, one familiar training loop. Everything in this guide is a descendant or rival of DPO.

First, a cheaper baseline: rejection sampling

Before reaching for any preference loss, know the simplest quality boost: rejection-sampling fine-tuning (also called best-of-N or RAFT). Sample several candidate responses from your SFT model, score them with a reward model, keep only the best, and fine-tune on those winners with an ordinary SFT loss. You are effectively distilling the model's own best behavior back into it.

DPO's weak spot, and IPO's fix

DPO has a subtle failure mode. Its loss keeps increasing the gap between the chosen and rejected response, with no natural stopping point. When a preference pair is near-deterministic (the chosen answer is almost always preferred), DPO will happily drive the probability of the rejected answer toward zero — and in doing so it can also crush the chosen answer's probability and distort the model, overfitting to the quirks of the preference dataset.

\mathcal{L}_{\mathrm{DPO}}(\theta)=-\,\mathbb{E}_{(x,y_w,y_l)}\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)}\right)\right]

The DPO objective: a logistic loss on the implicit reward margin between the chosen and rejected response — with no upper bound, the very gap that runs to infinity and that IPO targets.

IPO (Identity Preference Optimization) fixes this by adding a regularizer that pins the preference gap to a target margin instead of letting it run to infinity. The model is trained to make the chosen response preferred by a controlled amount, then stop — which prevents the degenerate collapse on near-deterministic data and makes IPO noticeably more robust to label noise. The cost is one extra hyperparameter (the target margin) to tune.

KTO: when you only have thumbs up/down

DPO and IPO both need paired data: for the same prompt, a chosen response and a rejected one. In the real world you usually have something messier and cheaper — a stream of single responses each tagged simply good or bad (a thumbs-up/down log). KTO (Kahneman–Tversky Optimization) is built for exactly this unpaired signal.

KTO borrows from prospect theory in behavioral economics: it models the utility of an output relative to a reference point, with separate weights for desirable and undesirable examples (people feel losses more than gains). Practically, it lets you align on abundant, naturally-collected feedback without the expensive chore of constructing matched pairs — a big deployment advantage when your signal is production telemetry rather than a curated annotation project.

ORPO: fold alignment into SFT

Every method so far assumes you already did SFT and now run alignment as a second stage — and most also need a frozen reference model held in memory. ORPO (Odds-Ratio Preference Optimization) collapses both stages into one. It adds a single odds-ratio penalty term on top of the ordinary SFT loss, so a single training run simultaneously teaches the model to imitate good responses and to disprefer bad ones — with no separate reference model at all.

\mathcal{L}_{\mathrm{ORPO}}=\mathcal{L}_{\mathrm{SFT}}-\lambda\,\log\sigma\!\left(\log\frac{\mathrm{odds}_\theta(y_w\mid x)}{\mathrm{odds}_\theta(y_l\mid x)}\right),\qquad \mathrm{odds}_\theta(y\mid x)=\frac{\pi_\theta(y\mid x)}{1-\pi_\theta(y\mid x)}

ORPO adds an odds-ratio preference penalty directly onto the SFT loss, so one stage with a single model in memory both fits and aligns — no separate reference model.

Choosing an alignment recipe

  1. Only thumbs up/down, unpaired? Use KTO — it is the only one here that does not require matched pairs.
  2. Want the leanest pipeline and have pairs? Use ORPO to do SFT and alignment in one stage with no reference model.
  3. Noisy or near-deterministic preferences? Prefer IPO over plain DPO — its margin regularizer resists the overfitting collapse.
  4. Have a strong reward model and generation budget? Iterate rejection sampling first; it is a robust, hard-to-beat baseline before any preference loss.
  5. Always evaluate on held-out preferences and real tasks, and watch for reward hacking and verbosity bias — alignment that games the metric is the central risk in this whole area of AI alignment.