human evaluation

Human evaluation puts real people in the loop to judge a model's outputs directly — rating how helpful, correct, fluent, or safe a response is, often on a scale or by choosing the better of two answers. It remains the gold standard, because the things we ultimately care about (Is this actually useful? Is it honest? Would I trust it?) are human judgments that no automatic metric fully captures.

The practical machinery matters. You need clear instructions, multiple raters per item to average out individual quirks, and a measure of how much they agree (inter-annotator agreement); low agreement means the question itself was ambiguous. Well-run human studies catch failures that benchmarks miss entirely — subtle dishonesty, tone, cultural fit, partial correctness.

The costs are real: it is slow, expensive, hard to reproduce exactly, and vulnerable to its own biases. Raters anchor on length and confidence, get tired, and bring cultural assumptions. So human evaluation is used where it counts most — final model comparisons and safety review — while cheaper automatic and model-graded methods cover the bulk, validated back against human ratings.