preference comparison

Ask a person 'on a scale of one to ten, how good is this answer?' and you get inconsistent, drifting numbers — one rater's seven is another's four. But ask 'which of these two answers is better, A or B?' and people are remarkably fast and consistent. A preference comparison is just that simple act: show a human two candidate behaviours and record which one they prefer. It is the atom out of which RLHF is built.

Each comparison yields a tiny fact: for a given context, output A was chosen over output B. These binary judgments are easier to give reliably than absolute scores, and they sidestep the need to calibrate a numeric scale across many different people. A reward model is then trained so that preferred items receive higher scalar scores, so a pile of 'A beats B' facts is quietly turned into a smooth, queryable reward function.

Comparisons are not free of trouble. People can prefer the wrong thing — longer, more confident, or more flattering answers — and they disagree with one another; rankings of more than two items grow unreliable fast. But as the basic unit of feedback, the relative judgment is the workhorse of preference-based learning, precisely because it asks of humans only what they are good at.