automatic text metrics

Before model-graded judging, the cheap way to score free-form text was to compare the model's output against one or more reference answers written by humans and measure word overlap. BLEU, built for machine translation, rewards matching short word sequences in the reference; ROUGE, built for summarisation, rewards covering the reference's content. Both are fast, deterministic, and need no human at scoring time, which made them the workhorses of text generation for years.

Their fatal weakness is that they only see surface form. A perfectly good paraphrase that uses different words scores poorly, while a fluent but wrong answer that happens to reuse reference words scores well. They cannot judge meaning, factual accuracy, or reasoning, and they assume the single reference is the only acceptable answer — false for most open-ended tasks.

Newer embedding-based metrics compare meaning rather than exact words and correlate better with human judgment, but the trend has moved toward LLM-as-a-judge and human ratings for anything open-ended. Overlap metrics survive as fast, reproducible sanity checks, not as verdicts on quality.