LLM-as-a-judge

For open-ended tasks like writing an email or explaining an idea, there is no single right answer to match against, so we ask a strong model to read the response and grade it. The judge model is given the question, the answer (or two competing answers), and a rubric — be helpful, accurate, well-structured — and it returns a score or picks a winner. It is the cheap, fast stand-in for a human rater, able to evaluate thousands of responses in minutes.

Done carefully it agrees with human judgments surprisingly often, which is why it now powers much of practical evaluation and even parts of training. But it inherits the judge's own biases and blind spots. Judges tend to prefer longer answers, the first option shown, responses in their own style, and confident wrong claims they cannot fact-check. They can be gamed by flattery or formatting.

So treat the judge as a calibrated instrument, not an oracle: fix the rubric, randomise answer order to fight position bias, swap which model judges, and spot-check against real human ratings. A judge that has never been validated against people is measuring its own preferences, not quality.