LLM-as-a-Judge and Preference Evaluation

Why open-ended outputs need a judge

Ask two chat models to 'explain entropy to a curious teenager' and there is no key to grade against. String-overlap metrics like BLEU and ROUGE were built for translation and summarisation and correlate poorly with human judgement of helpful, free-form answers. So we move to comparative evaluation: instead of scoring an answer in isolation, we ask which of two answers is better. Comparison is easier and more reliable for graders than absolute scoring — the same reason RLHF collects pairwise preferences rather than numeric ratings.

Win-rate: the pairwise primitive

Win-rate is the simplest comparative metric: run model A and a fixed baseline B on the same prompts, have a grader pick the winner of each pair, and report the fraction of prompts where A wins. A 50% win-rate against the baseline means parity; 70% means A is preferred on most prompts. It is intuitive and bounded, but it depends entirely on which baseline and which prompts — a 90% win-rate against a weak baseline is nearly meaningless. Length is a notorious confound: graders tend to prefer longer answers, so always check whether a win-rate gain survives length control.

Crowd preference at scale: the Elo arena

Chatbot Arena turns win-rates into a global ranking. Anonymous users type a prompt, receive two responses from undisclosed models, vote for the better one, and only then see the names. Aggregating millions of these pairwise votes with an Elo (or Bradley–Terry) rating yields a leaderboard grounded in real human preference on real prompts — its great strength, since no static benchmark covers what people actually ask.

The Elo model: a rating gap sets the expected win probability, and each vote nudges the ratings.

Using a model as the judge

Human votes are slow and costly, so we replace the grader with a strong model: LLM-as-a-judge. Give a capable model the prompt, two candidate answers, and a grading instruction; it returns a verdict and a rationale. On well-specified tasks, a strong judge agrees with human majority vote roughly as often as two humans agree with each other — cheap enough to grade thousands of comparisons per hour and the engine behind automated leaderboards like MT-Bench and AlpacaEval. The same machinery, scaled, is how we approach scalable oversight of systems too large to grade by hand.

judge_prompt = f"""You are grading two assistant answers to the same question.
Question: {q}
[A]: {answer_a}
[B]: {answer_b}
Decide which answer is more helpful and correct.
First reason step by step, then end with exactly: VERDICT: A | B | TIE"""
# Mitigation: run twice with A/B swapped; keep the verdict only if it is
# consistent across both orders, else record TIE (controls position bias).

The judge has biases

A model judge is itself a model, so it inherits failure modes. The most studied is position bias: presented with the same two answers, many judges systematically prefer whichever appears first (or sometimes second). Others include verbosity bias (longer looks better), self-preference (a judge favours text in its own style or from its own family), and sycophancy (agreeing with an assertion in the prompt). Left uncontrolled, these biases can swamp the real quality difference you are trying to measure.

Averaging the judge's verdict over both presentation orders cancels first-order position bias.

1. Swap A/B order and average, or keep only order-consistent verdicts, to cancel position bias.
2. Control for length: report win-rate at matched answer lengths, or use a length-debiased estimator.
3. Calibrate the judge against a few hundred human labels and report judge–human agreement as a number.
4. Use a different model family as judge than the ones under test to blunt self-preference.

Rubrics: making judgement legible

Vague instructions give noisy verdicts. Rubric-based evaluation replaces 'which is better' with explicit criteria — factual accuracy, completeness, safety, format adherence — each scored separately against a written standard, often with reference points for what a 1, 3, and 5 look like. Rubrics raise agreement between graders, expose why one answer wins, and let you weight what matters for your application. They are the bridge between the preference culture of this guide and the disciplined, auditable suites you will build in Guide 5.