Why this rung is different
Everything earlier in this rung was about making a large language model *produce* something — a sharper prompt, a reasoning chain, a retrieval step to ground the answer. This final guide is about the question that decides whether any of it ships: *is the output actually good, and what happens when it isn't?* That sounds dull next to clever prompting, but it is where real systems live or die. A demo that wows you once is not a product; a product is something you can trust on the ten-thousandth query you never personally read.
Two facts make evaluating generative output genuinely hard. First, there is rarely one right answer — a summary, an email, or a code fix can be excellent in a dozen different shapes, so the crisp accuracy number you used for a classifier doesn't apply. Second, the model is fluent even when it is wrong: it will state a confidently-worded falsehood in the same warm tone it uses for the truth. So we need new tools — evals built for open-ended text, and guardrails that assume the model will sometimes misbehave and catch it anyway.
Evals: measuring open-ended output
An eval is just a repeatable test of model quality: a fixed set of inputs, a way to score the outputs, and a number you can watch over time. The discipline is the same one you met when you learned about a benchmark and the value of a baseline — you cannot improve what you cannot measure. The catch for generative text is the *scoring*, and there are three honest families of approaches, each with real limits.
- Reference-based metrics. Compare the output against a gold answer with measures like BLEU, ROUGE, or embedding similarity. Cheap and automatic — but they reward surface overlap, so a correct paraphrase can score low and a fluent wrong answer can score high.
- LLM-as-judge. Ask a strong model to grade the output against a rubric ("is it faithful to the source? is it on-topic?"). It scales to open-ended text — but the judge has its own biases (it favors longer, more confident answers) and can be gamed, so spot-check it against humans.
- Human evaluation. People rate or rank outputs, sometimes via head-to-head A/B comparisons. It is the gold standard for taste and safety — but it is slow, expensive, and noisy, so you reserve it for the cases the cheap methods cannot judge.
Whichever you use, the most valuable artifact you build is not the metric — it is the eval set itself: a curated collection of real, hard, representative inputs with notes on what a good answer looks like. Grow it every time the system fails in production; each bug becomes a permanent test. This is the same instinct as a held-out test set in classic machine learning, and the same warning applies: guard it against leakage, because the day your prompt is secretly tuned to pass your own eval, the number stops meaning anything.
Hallucination: confident, fluent, and wrong
A hallucination is when the model states something false or unsupported as if it were fact — a fake citation, an invented API, a confidently wrong date. It is tempting to call this a bug to be patched, but it is more honest to see it as inherent to how these models work. An LLM is trained by predicting the next token; it learns the *shape* of plausible text, not a database of verified truths. Generating a smooth, likely-sounding sentence is exactly its job — and a smooth lie is just as likely-sounding as a smooth fact.
So how do you fight it? You reduce it, you don't eliminate it. The single biggest lever is grounding the model in real sources with retrieval-augmented generation and then *checking that the answer actually traces back to those sources* — a faithfulness eval, not just a vibe. You can lower temperature so the model commits to its most likely (and usually safest) continuation instead of a creative tangent. And you can ask the model to cite, to say "I don't know," or to expose its reasoning so a checker — human or automated — has something to verify against.
Guardrails: catching trouble at the edges
A guardrail is a check that sits *around* the model — at the input or the output — rather than inside it. The mental model is a factory line: the model is the worker, and guardrails are the inspectors before and after. On the way in, you screen for prompt injection (a user or a retrieved document trying to overwrite your system prompt), off-topic requests, or attempts to extract secrets. On the way out, you check the response before it ever reaches the user: does it leak personal data, contain disallowed content, or break the required format?
user input │ ▼ [ input guardrail ] ──blocked──► refuse / safe reply │ ok ▼ LLM ◄── system prompt + retrieved context │ ▼ [ output guardrail ] ──flagged──► block / redact / escalate │ ok ▼ response to user
Guardrails come in layers, and the cheap layers go first. Simple rules — regular expressions, blocklists, a JSON-schema validator, a length cap — catch a surprising amount for almost no cost or latency. Above them sit small classifiers and moderation models that flag toxicity or self-harm content. At the top, a second LLM call can judge nuance the rules miss. The art is ordering them so the fast, certain checks run before the slow, fuzzy ones — and remembering that every guardrail adds latency and cost, so you spend that budget where the risk actually is.
The human in the loop
No eval and no guardrail is perfect, so the last layer of safety is a person. The human-in-the-loop pattern keeps a human in the decision path — reviewing, approving, or correcting — especially where a wrong answer is costly or irreversible. The right amount of human is a dial, not a switch. A throwaway brainstorm needs none; an email draft wants a glance before it sends; a medical or legal suggestion, a refund over a threshold, or anything an agent does that touches the real world should pause for a human to approve. Match the friction to the stakes.
Human review is also how the system gets better over time. Every correction is a labelled example: feed it back into your eval set, into your guardrail rules, or — if a pattern is large and stable enough — into the data you'd use to fine-tune. This closes the loop you have been building across the whole rung. Prompting and retrieval shape the output; evals and guardrails measure and contain it; humans catch what slips through and teach the system what 'good' means. That is what it takes to move from a clever demo to something you can actually ship and stand behind.