win-rate evaluation

A win rate is the fraction of prompts on which a model's output is judged better than a fixed reference model's. Instead of an absolute score per answer, you compare head-to-head against a baseline — say GPT-4 or a previous checkpoint — and report that model X beats the baseline 62% of the time. It is the metric behind AlpacaEval and most instruction-tuning comparisons.

A judge, human or LLM, sees both answers per prompt and picks a winner (ties split or counted separately); averaging the indicator over the eval set gives the win rate with a binomial confidence interval. Because raw win rates are inflated by length and style, length-controlled variants such as LC-AlpacaEval regress out response length so a model cannot climb simply by writing more. Win rate is ordinal and relative: it tells you the ranking against this baseline, not by how much in any absolute sense.

The number is meaningless without naming the baseline and the judge, and it is intransitive — A beating B and B beating C does not guarantee A beats C. It also conflates being right with being preferred, so on tasks with verifiable answers an exact-match or execution metric is more trustworthy than a preference win rate.