evaluating LLMs

Evaluating a language model means asking, in a fair and repeatable way, how good it actually is. The same model can write a lovely poem and then confidently invent a fake citation, so a single chat never tells you enough. Instead we run it through many controlled tasks — answering exam questions, solving math, writing code, summarising documents — and count how often it gets things right. The goal is a number you can trust and compare across models, not a vibe from one lucky conversation.

Good evaluation separates three things. Capability: can the model do the task at all? Reliability: does it do it consistently, not just sometimes? Alignment: does it stay helpful, honest, and safe while doing it? Each needs different tools — automatic scoring against known answers, head-to-head comparisons, human ratings, or another model acting as judge. No single method is complete, so serious evaluation stacks several and reports the gaps.

The honest catch is that we mostly measure what is easy to measure. Multiple-choice accuracy is cheap; judging whether a long essay is genuinely insightful is not. So benchmarks always lag real-world usefulness, and a high score is evidence of skill, never proof of it.