position bias in LLM judges

When an LLM judge compares two answers, it is not neutral about where they sit. Position bias is its systematic tendency to favor whichever answer appears first (or sometimes last) regardless of content — present the same pair in the opposite order and the verdict can flip. It is one of several judge pathologies, alongside verbosity bias (longer looks better) and self-preference (a model favors text in its own style).

The bias arises because the judge's preference is entangled with the prompt's token layout rather than purely with answer quality. The standard diagnostic and fix is order randomization with consistency checking: evaluate each pair in both orders, (A,B) and (B,A); count a clear win only when the judge agrees both ways, and treat order-dependent verdicts as ties. Reporting the swap-consistency rate quantifies how much of a result is signal versus position artifact; calibration and few-shot debiasing help further.

Position bias does not vanish with stronger judges — it shrinks but persists — and it interacts with the others, so a verbose answer placed first compounds two biases at once. Any win-rate or arena-style result that does not randomize order and report consistency should be read with suspicion, because half its apparent signal may be layout.