direct preference optimization (DPO)

The standard RLHF pipeline is a two-step dance: first train a reward model from preferences, then run RL against it. DPO asks a sharper question — what if we skip the middleman entirely? It turns out you can rearrange the maths so the policy is optimised directly on the preference pairs, with no separate reward model and no RL loop at all: just a supervised-style loss on 'preferred over rejected' data.

DPO starts from the very objective of KL-regularised RLHF — maximise reward minus KL from a reference — and uses the known closed form of its optimal policy to express the implicit reward in terms of the policy itself. Substituting that into the Bradley-Terry preference likelihood collapses everything into a simple loss: raise the policy's log-probability of preferred responses relative to rejected ones, each measured against the frozen reference. The policy quietly becomes its own reward model.

DPO is dramatically simpler and more stable than PPO-based RLHF — no reward model to train, no rollouts, no value head — and has become a popular default for preference tuning. But it is fully offline, learning only from a fixed set of comparisons, so it cannot explore for genuinely new high-reward behaviour the way online PPO can, and it can still overfit to the quirks and biases baked into its preference data.