pairwise comparison

Instead of asking a rater to give an answer an absolute score out of ten, pairwise comparison shows two answers to the same prompt and asks the simpler question: which one is better? People are far more reliable at relative judgments than absolute ones — it is hard to say whether an essay deserves a 7, but easy to say this one beats that one. Aggregating many such votes gives a win rate: the fraction of match-ups a model wins.

This format powers both human preference studies and model-graded evaluation, and it is the raw material behind preference-trained models, where the winning answer teaches the model what people like. It is robust to scale drift, since you only ever compare, never calibrate a number.

Its weaknesses are subtle. Which answer is shown first sways the vote (position bias), so order must be randomised or both orders averaged. A win rate also tells you that A beats B, not by how much, and a model can win narrowly on everything yet excel at nothing. To turn many pairwise results into a single ranking you usually feed them into an Elo or Bradley–Terry model.