KTO (Kahneman–Tversky optimization)

Most alignment methods need preference pairs: for the same prompt, a chosen answer and a rejected one. That data is expensive and often unnatural, because real feedback usually arrives as isolated judgments — this output was good, that one was bad — not as matched comparisons. KTO removes the pairing requirement entirely: it trains on individual outputs each carrying just a binary label, desirable or undesirable, which is the kind of signal you can harvest cheaply from thumbs-up and thumbs-down logs.

The objective borrows the shape of human value from Kahneman and Tversky's prospect theory. Rather than maximizing a preference probability, KTO measures each output's implied reward relative to a reference point — roughly the model's average behavior — and passes it through a concave-for-gains, convex-for-losses value function with built-in loss aversion. Desirable examples are pushed up and undesirable ones pushed down, but asymmetrically, so the model is penalized harder for bad outputs than it is rewarded for good ones, mirroring how people actually weigh risk.

In practice KTO matches or beats paired methods like DPO at comparable scale while feeding on much messier, more abundant data, and it tolerates an imbalance between positive and negative examples. The price is two extra hyperparameters governing the loss-aversion asymmetry, and a reliance on the prospect-theory framing being a good model of what you want the system to value.