JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Collecting preference data

The whole pipeline is only as good as its comparisons. How to design the pairs, choose who labels, and when to let an AI do the judging.

What one preference record looks like

Before designing a collection process, picture a single row of the dataset. A comparison is a context, two responses to that context, and a label saying which response a human chose. That is the entire atom of human preference data.

{
  "context": "Summarize: <article text>",
  "response_a": "<summary A>",
  "response_b": "<summary B>",
  "preferred": "a"        // the human picked A over B
}
One comparison. The model never sees a number — only the winner of a pair.

For control tasks the two responses become two trajectory clips — say, two short videos of a robot reaching for a cup — and the rater picks the clip closer to the goal. The shape is identical; only the medium changes.

Designing the collection

Good preference dataset collection is mostly about which pairs you show. Random pairs waste rater time on obvious wins; the most informative pairs are close calls where the current model is genuinely unsure. Many teams sample several responses from the model and ask raters to compare the most ambiguous ones — a light form of active learning.

  1. Fix the contexts: a representative set of prompts or starting states you actually care about.
  2. Generate candidates: sample two or more responses per context from the current policy.
  3. Pair and present: show raters two at a time, ideally close in quality.
  4. Record the winner — and optionally a tie or "both bad" escape hatch so raters aren't forced into a false choice.
Everything in this guide builds stage one: the comparisons you collect become the data that trains the reward model and, through it, the policy.

RLHF pipeline diagram in which the human-preference collection stage is the first step feeding the reward model and then the policy.

Who labels, and keeping it clean

The quality of preference annotation sets a ceiling on everything downstream. Raters need a written guideline that says what "better" means for this task — more accurate? safer? more concise? — because their private definitions will otherwise diverge. Disagreement among raters is information, not just noise: high-disagreement pairs often reveal an ambiguous instruction or a genuinely hard trade-off.

Because raters are humans in the loop, watch for the usual hazards: position bias (always picking the first option), length bias (longer looks more thorough), and fatigue. Randomize which response is shown as "A", audit a fraction of labels, and measure inter-rater agreement as a health check.

When an AI does the judging

Human comparisons are expensive and slow. A powerful alternative replaces some human judges with an AI judge guided by a written set of principles — constitutional feedback, the idea behind Constitutional AI. A model critiques and ranks responses against an explicit "constitution" (be helpful, refuse harmful requests, and so on), producing comparisons at scale.

When the labels come from a model rather than a person, the technique is called RLAIF — reinforcement learning from AI feedback. It trades a little fidelity for a lot of throughput, and in practice teams blend the two: scarce human comparisons to anchor what matters, abundant AI comparisons to fill in the rest.

An AI judge in place of a person: an LLM reasons over written principles to pick the better response, the heart of RLAIF.

Diagram of an LLM loop reasoning, acting, and observing, here used to judge responses.