pairwise preference comparison
At the optician, you do not try to grade your vision out of ten; the doctor flips between two lenses and asks 'one, or two?'. Repeating that simple choice many times pins down your prescription precisely. Pairwise preference comparison is exactly this method applied to AI answers: show a person two candidate responses to the same prompt and ask which is better. It is the raw data that feeds reward models and RLHF.
The workflow is straightforward. For a given prompt, the model produces two (sometimes more) answers; a human annotator reads both and marks a winner, optionally noting how strong the preference is. Collected by the thousands, these (prompt, chosen, rejected) triples become the training set: a reward model or a direct-preference method learns to make the chosen answer outscore the rejected one. Pairs are used rather than 1-to-10 scores because a binary 'this one is better' is faster, less noisy, and more consistent across different annotators.
The honest caveat is that the comparisons are only as good as the people and process behind them. Annotators disagree, get tired, and pick up on superficial cues, famously rating longer or more confident answers higher even when they are not actually better. These biases pass straight into the reward model and then into the trained policy, which is one root of problems like sycophancy and length inflation. Careful guidelines, multiple annotators per pair, and quality checks all help, but never fully remove the noise.
Prompt: 'Explain photosynthesis to a child.' Answer A is a warm two-sentence analogy; answer B is a dense paragraph of jargon. The annotator clicks A, creating one (prompt, chosen=A, rejected=B) training pair.
A single 'which is better?' click becomes one labelled preference pair.
Watch for length and confidence bias: raters often pick the longer or more assertive answer regardless of correctness, and that bias is exactly what the reward model then learns to chase.