human preference data

Human preference data is the fuel of RLHF. Rather than asking people to write ideal answers, which is slow and inconsistent, you show them the same prompt with two different model responses and ask the easier question: which one is better? Collected at scale, these pairwise comparisons encode a lot about what people find helpful, clear, safe, and honest, without anyone having to spell out the rules.

Each data point is roughly a triple: a prompt, a chosen response, and a rejected response. The choices may come from different models, different sampling settings, or different decoding temperatures, so the dataset spans a range of quality. A reward model is then trained to give the chosen response a higher score than the rejected one, learning the geometry of human preference from many such comparisons.

The quality of this data caps everything downstream. Vague guidelines, rushed raters, or annotators who reward confident-sounding but wrong answers all bake quiet biases into the reward model, and the final policy inherits them.