LLM-as-a-judge

Instead of asking a human to score every answer, you hand a transcript and a rubric to a strong model and ask it to grade — pick the better of two answers, or assign a 1 to 10 score. It is cheap, fast, and reproducible enough to evaluate thousands of samples per hour, which is why it underpins most modern open-ended LLM evaluation where there is no single reference answer to match against.

Two protocols dominate: pairwise (which of A or B is better, optionally allowing ties) and single-answer (score one response against criteria). The judge is prompted with the question, the candidate output or outputs, and an explicit rubric; asking it to reason before stating a verdict raises agreement with humans. Reported agreement on chat tasks often exceeds 80%, rivaling inter-annotator agreement — but the judge inherits biases: it favors the first-shown answer, longer answers, text in its own style, and surface fluency over factual correctness.

Treat the judge as a noisy instrument, not an oracle: calibrate it against a human-labeled subset, randomize answer order and average over both, and avoid letting a model grade its own family unguarded. It is a proxy for quality, useful precisely because it is cheap to run at scale — not a substitute for ground truth on tasks where correctness can be checked directly.