preference learning

Suppose I ask you to rate a song from 1 to 10. Your number wobbles with your mood, and your 7 may be my 5. But if I play you two songs and ask which you like more, your answer is far steadier and easier to give. Preference learning is the idea of teaching a system from these relative comparisons (A is better than B) rather than from absolute scores, because people are much more reliable at comparing than at grading.

Concretely, you gather a pile of comparisons, each one a prompt with a winning answer and a losing answer, and you fit a model that assigns every answer a hidden score such that the winners tend to score higher than the losers. In RLHF this fitted scorer is the reward model, and the standard recipe for converting 'A beat B' into numerical scores is the Bradley-Terry model. The same idea also powers reward-model-free methods like direct preference optimization, which learn straight from the comparison pairs.

Preference learning matters for safety because it is how we communicate fuzzy human wishes (be helpful, do not be rude, do not help build weapons) that we cannot write as explicit rules. But it carries an honest caveat: a preference dataset encodes whose preferences were collected and under what instructions. Annotator disagreement, cultural bias, and quirks like preferring longer or more confident answers all flow into the learned objective, so 'what people preferred' is not automatically 'what is correct or good'.