IPO (identity preference optimization)

DPO works by turning a preference dataset directly into a loss, but it inherits a hidden assumption from its logistic form: it treats the implied reward gap between a chosen and a rejected answer as something to be pushed ever larger. When a pair is always labeled the same way — a deterministic or near-deterministic preference — that pressure never stops, the model drives the reward gap toward infinity, and the constraint meant to keep it near the reference model effectively evaporates, inviting overfitting.

IPO diagnoses this and patches it with a different objective. Instead of the unbounded logistic loss, it regresses the difference in implied log-probability ratios toward a fixed finite margin, using a squared loss. Because the target is a constant rather than infinity, the model is content once the chosen response leads the rejected one by that set margin and stops there; the reference-anchoring regularization keeps its grip even on perfectly consistent data. It is, in effect, DPO with the runaway removed.

The practical reading is that IPO is more robust when preferences are clean, sparse, or strongly deterministic — exactly the regimes where DPO quietly overfits. The cost is one more hyperparameter, the margin, and the fact that on noisy, abundant preference data the difference from DPO can shrink, since there the logistic loss rarely gets the chance to run away in the first place.