ORPO (odds-ratio preference optimization)

The usual recipe for aligning a model runs in two stages: first supervised fine-tuning to teach the format, then a separate preference step like RLHF or DPO to teach taste, the latter typically requiring a frozen reference copy of the model. ORPO collapses this into a single stage with no reference model at all. It fine-tunes on the chosen responses while, in the very same loss, discouraging the rejected ones — one pass, one model in memory, no second phase.

The mechanism is an odds-ratio penalty added to the standard supervised likelihood. Alongside the usual next-token loss on the preferred answer, ORPO adds a term that increases the odds of generating the chosen response relative to the odds of generating the rejected one. Using the odds ratio rather than a raw probability ratio keeps the penalty gentle, so it nudges the model away from disfavored outputs without the runaway divergence that plagues some reference-free objectives, all while the supervised term keeps the model fluent.

The appeal is operational simplicity and lower cost: no reward model, no reference model, no extra stage, just a slightly modified fine-tuning loss with one weighting hyperparameter. That same minimalism is the caveat — without a reference anchor the strength of the preference penalty must be tuned carefully, and ORPO is most natural when you already have paired preference data to fold into the fine-tuning set.