preference dataset collection

A reward model is only as wise as the comparisons it learns from, so gathering those comparisons at scale is half the battle of RLHF — and far more of a human-factors and engineering problem than it first looks. You need prompts that cover the situations you actually care about, model outputs to compare, trained annotators to judge them, and guidelines so that different people judge in roughly the same way.

A typical loop samples prompts from real usage or curated sets, generates two or more candidate responses per prompt from the current model, and presents them to annotators who pick the better one against a rubric. Quality control matters enormously: rater training, inter-annotator agreement checks, gold-standard questions, and down-weighting unreliable labellers. The data is deduplicated and split, and often refreshed as the policy improves, since yesterday's comparisons drift out of the distribution the agent now visits.

Every bias in this pipeline becomes a bias in the model. Annotator demographics, fatigue, instructions that quietly reward length or confidence, and the choice of prompts all leave fingerprints on the final behaviour. Collecting preference data well is less about sheer volume and more about coverage, consistency, and honest rubrics — get those wrong and no amount of RL will fix the result.