DPO vs RLHF

Direct preference optimization, DPO, is a shortcut that gets much of what RLHF gets without the reinforcement-learning machinery. The insight is mathematical: the optimal policy of the KL-penalized RLHF objective can be written directly in terms of the reward, so you can fold the reward model away and turn the whole thing into a single supervised-style loss on preference pairs. There is no separate reward model and no sampling loop.

Concretely, DPO trains the model so that, for each pair, it raises the likelihood of the chosen response and lowers the likelihood of the rejected one, measured relative to a frozen reference model. The reference and an implicit KL term are baked right into the loss, so the model still stays close to its starting point. The result is a stable, easy-to-run procedure that often matches PPO-based RLHF on common benchmarks.

The trade-offs are real. RLHF can sample fresh responses and get reward on its own outputs, which helps it explore; DPO only ever sees the fixed preference pairs in the dataset. DPO can also be more prone to drifting in odd ways when the data is narrow. Which wins depends on the data, the budget, and how much online exploration the task needs.