preference optimization methods

Once DPO showed you could align a model directly from preference data without an RL loop, a whole family of related objectives followed. They share a goal — push chosen responses above rejected ones while staying near a reference model — but differ in the exact loss, and each fixes a perceived weakness of the others.

IPO adds a term that limits how hard the model pushes on any single pair, guarding against the overfitting DPO can show when a preference is near-deterministic. KTO drops the need for paired data entirely: it learns from individual responses labelled simply good or bad, using ideas from prospect theory, which is handy when real-world feedback arrives as thumbs-up and thumbs-down rather than clean comparisons. ORPO folds preference learning into the supervised fine-tuning step itself, removing the separate reference model.

There is no universal winner. The right choice depends on what your data looks like — paired or unpaired, clean or noisy, abundant or scarce — and on how much you worry about over-optimization. The shared lesson is that much of RLHF's benefit can be captured by careful offline objectives.