direct preference optimization

Classic RLHF is a three-part machine: train a reward model, then run a reinforcement-learning loop (PPO) that uses it, all while juggling a KL penalty. That is powerful but fiddly, with extra models in memory and many tuning knobs. Direct preference optimization is a cleverly simpler route to roughly the same destination: train the language model directly on the preference pairs, skipping the separate reward model and the RL loop entirely.

The trick rests on a piece of mathematics. The policy that RLHF is trying to find (maximise reward while staying close to the base model via a KL penalty) has a known closed-form relationship to the reward. DPO turns that relationship around: it rewrites the objective so the reward is expressed in terms of the policy itself, leaving a simple loss you can optimise with ordinary supervised-style training. In effect you nudge the model to raise the probability of each chosen answer and lower the probability of each rejected answer, by an amount tied to the same Bradley-Terry logic and with an implicit KL leash still built in. One model, one stable training loop, no reward model to over-optimize separately.

DPO and related reward-model-free methods became popular because they are simpler, cheaper, and more stable to train than PPO-based RLHF, while often reaching comparable quality, which matters for smaller teams. The honest caveats: DPO still learns from a fixed preference dataset, so it inherits the same label biases and is still optimising a proxy for human preference, just an implicit one; and there is ongoing debate about whether it truly matches well-tuned online RLHF, especially at the frontier and on the hardest safety behaviours. It changes the machinery, not the fundamental limits of learning from preferences.