preference data annotation

Preference data annotation is the human craft underneath RLHF: the process by which real raters look at two responses and decide which is better. It sounds simple, but it is where a lot of a model's eventual character is quietly set. Raters work from guidelines that define what to value — accuracy, helpfulness, safety, tone — and how to weigh them when they conflict.

Good annotation is hard for human reasons. Raters get tired and default to surface cues like length, confidence, or formatting; long answers look thorough even when padded; agreeable answers feel better even when wrong. Different cultures and individuals disagree about borderline cases. Programs fight this with clear rubrics, calibration examples, inter-rater agreement checks, and sometimes multiple raters per comparison.

Because the reward model can only learn what the annotations teach it, every systematic bias here — rewarding confident wrongness, over-rewarding refusals, favoring verbosity — becomes a bias the deployed model inherits. The annotation guidelines are, in a real sense, a draft of the model's values.