RAG evaluation

A RAG system can fail in two quite different places. It can retrieve the wrong passages, in which case even a perfect model is doomed; or it can retrieve exactly the right passages and still answer badly. Because of this, you have to measure both halves separately — not just glance at whether the final answer happens to look good.

Retrieval is scored with ranking metrics like recall and precision at k: did the passage that holds the answer make it into the shortlist at all? Generation is scored on faithfulness — does every claim in the answer actually follow from the retrieved context? — and on answer relevance, whether it addresses the question that was asked. Faithfulness is often judged by a second LLM acting as a grader. The reason to split these out is practical: bad retrieval and bad generation call for completely different fixes.