RAG Evaluation

Learn how to evaluate RAG systems: retrieval metrics, generation metrics, golden datasets, automated and LLM-as-a-judge evaluation, human review and failure analysis.

25 minIntermediateCode Examples
The Core Idea: A RAG demo that works on five handpicked questions proves very little. Evaluation replaces impressions with numbers: does retrieval find the right evidence, does the answer stay faithful to it, and does it actually address the question? Because RAG has two stages that can fail independently, a good evaluation measures both separately, then points to what to fix.

1. Beginner: Why RAG Needs Its Own Evaluation

When a RAG answer is wrong, the cause may lie in retrieval, in generation, or in the documents themselves. Judging only the final text hides which.
What went wrongFinal answer looks likeWhere to fix
The right chunk was never retrievedWrong or vagueChunking, embeddings, retrieval
The right chunk was retrieved but ignoredWrong, though the evidence was therePrompt, model, context order
The model added facts of its ownFluent but unsupportedGrounding instructions, verification
The source document is wrongFaithful but incorrectData quality
Rule: evaluate the stages separately first, then the whole system. A single score for the whole pipeline cannot tell you what to change.

2. Intermediate: Retrieval Evaluation

These measures compare what was retrieved with what should have been. You already know precision and recall from the retrieval lesson. Two additional measures are especially useful for RAG.
Context precision
What share of the retrieved chunks are relevant? Low values mean noise in the prompt.
Context recall
What share of the facts needed for the answer appear anywhere in the retrieved context? Low values mean the answer cannot be produced.
context recall=needed facts found in the retrieved contextall needed facts\text{context recall} = \dfrac{\text{needed facts found in the retrieved context}}{\text{all needed facts}}
Other retrieval measures include hit rate, recall@k and MRR. Context recall is the most direct signal that retrieval is the bottleneck.

3. Intermediate: Generation Evaluation

Generation quality has three separate aspects, and an answer can pass one while failing another.
AspectQuestion it answersCompared against
Answer relevanceDoes the answer actually address the question?The question
Faithfulness, or groundednessIs every claim supported by the retrieved context?The retrieved context
Correctness, or factualityIs the answer true?A reference answer or trusted facts
faithfulness=claims supported by the contextall claims in the answer\text{faithfulness} = \dfrac{\text{claims supported by the context}}{\text{all claims in the answer}}
Faithful is not the same as correct: if the retrieved document is wrong or outdated, a perfectly faithful answer is still wrong. That is why correctness needs a reference, and why data quality is part of evaluation.

4. Intermediate: Golden Datasets and Evaluation Methods

A golden dataset is a set of questions with known good answers and the sources that support them. It is the ruler against which you measure every change.
Collect real questions from users or logs, covering common cases, rare cases and questions with no answer.
Label the needed source chunks and a short reference answer, checked by someone who knows the topic.
Extend with synthetic questions generated by a model from your documents, but review them, since generated questions can be too easy or unnatural.
Maintain the set as documents and user needs change.
MethodStrengthWeakness
Automated metricsFast, cheap, repeatableRough, can miss meaning
LLM-as-a-judgeScales, understands language, follows a rubricCan be biased toward long or first-listed answers and must be checked against humans
Human evaluationMost trusted, catches subtle problemsSlow and costly, so use it on samples and high-stakes cases
When using an LLM-as-a-judge, give it a clear rubric, ask for a reason before the score, and compare its scores with human labels on a sample. A judge you have not calibrated is an opinion, not a measurement.

5. Advanced: Failure Analysis and the Improvement Loop

Numbers tell you how well the system performs. Failure analysis tells you why. For each failed question, walk the pipeline from left to right and stop at the first broken stage.
Was the source in the knowledge base at all? If not, it is a data gap.
Was it retrieved? If context recall is low, look at chunking, embeddings, filters and query transformation.
Was it ranked high and included? If not, look at reranking and context selection.
Did the answer use it faithfully? If not, look at prompts, ordering and verification.
Was the answer complete and relevant? If not, look at instructions and answer format.
Make it routine: run the golden set after every change, keep the history of scores, and treat a drop as a failing test. Remember that small test sets are noisy, so a change of a point or two may be chance. Look for consistent improvement and examine the individual questions that changed.

6. Visual Explanation: Measuring Both Halves of RAG

Panel A separates retrieval measures from generation measures. Panel B shows the scores from the code below on four test questions. Panel C is a diagnosis explorer.
Panel A · Two Halves, Different QuestionsMeasure each half on its own
Retrieval
Did we find the evidence?
Context precision
↓
Context recall
↓
Low recall: fix search and chunks
Generation
Did we use it well?
Answer relevance
↓
Faithfulness and correctness
↓
Low faithfulness: fix prompt and checks
Panel B · Results from the CodeFour golden questions
QuestionContext recallFaithfulnessCorrectDiagnosis
1. refund time1.01.001.0pass
2. shipping time1.01.001.0pass
3. premium shipping0.00.140.0retrieval failure
4. password length1.00.750.0generation failure
Average correctness is 0.5, but the table shows that two different problems cause it
Panel C · Diagnosis ExplorerClick a symptom to open it
Click to see why averages hide problems▼
An average correctness of 0.5 could mean every answer is half right or that half the answers are perfect and half are wrong. Always look at the failing cases, which is where the fixes are.
How to read it: Panel A names the two halves, Panel B shows that different questions fail for different reasons, and Panel C turns a symptom into the next place to look.

7. Python Implementation: A Small RAG Evaluation Harness

The program holds five chunks and a golden dataset of four questions. Each entry records the chunks that should be retrieved, the facts the answer needs, and what the system under test actually retrieved and answered. For every question it computes context precision, context recall, a word-overlap faithfulness score, answer correctness and a rule-based judge verdict, then assigns a failure type by walking the pipeline from left to right.
rag_evaluation.pyPython 3.10+ · Standard library only
chunks = dict([    ("c1", "Refunds are issued within 5 business days."),    ("c2", "Standard shipping takes 5 to 7 business days."),    ("c3", "Premium members get free shipping."),    ("c4", "Support is open Monday to Friday."),    ("c5", "Passwords must contain 12 characters."),]) stop_words = set(["the", "a", "is", "are", "to", "of", "for", "must", "do", "does", "how", "many", "long"]) # Golden dataset plus what the system under test producedcases = [    ("How long do refunds take?", ["c1"], ["5 business days"], ["c1", "c4", "c2"], "Refunds are issued within 5 business days."),    ("How long is shipping?", ["c2"], ["5 to 7 business days"], ["c3", "c2", "c4"], "Standard shipping takes 5 to 7 business days."),    ("Do premium members pay for shipping?", ["c3"], ["free shipping"], ["c2", "c1", "c4"], "Premium members pay a flat fee of 5 dollars."),    ("How many characters must a password have?", ["c5"], ["12 characters"], ["c5", "c4", "c1"], "Passwords must contain 16 characters."),] def words(text):    cleaned = "".join(ch if ch.isalnum() else " " for ch in text.lower()).split()    return set(w for w in cleaned if w not in stop_words) def faithfulness(answer, evidence):    a = words(answer)    return len(a & words(evidence)) / len(a) def judge(answer, evidence):    # a rule-based stand-in for an LLM judge    return "faithful" if faithfulness(answer, evidence) >= 0.8 else "unfaithful" totals = [0.0, 0.0, 0.0, 0.0, 0.0]print("Q | ctx precision | ctx recall | faithfulness | answer correct | verdict")for number, (question, gold_ids, facts, retrieved, answer) in enumerate(cases, start=1):    evidence = " ".join(chunks[c] for c in retrieved)    precision = len(set(gold_ids) & set(retrieved)) / len(retrieved)    context_recall = sum(1 for f in facts if f in evidence) / len(facts)    faith = faithfulness(answer, evidence)    correct = sum(1 for f in facts if f in answer) / len(facts)    if context_recall < 1:        verdict = "retrieval failure"    elif correct < 1:        verdict = "generation failure"    else:        verdict = "pass"    for i, value in enumerate([precision, context_recall, faith, correct]):        totals[i] += value    print(number, "|", round(precision, 2), "|", context_recall, "|", round(faith, 2), "|", correct, "|", verdict, "| judge:", judge(answer, evidence)) n = len(cases)print("Average ctx precision:", round(totals[0] / n, 2), "| ctx recall:", round(totals[1] / n, 2), "| faithfulness:", round(totals[2] / n, 2), "| answer correct:", round(totals[3] / n, 2))
Expected output
Q | ctx precision | ctx recall | faithfulness | answer correct | verdict1 | 0.33 | 1.0 | 1.0 | 1.0 | pass | judge: faithful2 | 0.33 | 1.0 | 1.0 | 1.0 | pass | judge: faithful3 | 0.0 | 0.0 | 0.14 | 0.0 | retrieval failure | judge: unfaithful4 | 0.33 | 1.0 | 0.75 | 0.0 | generation failure | judge: unfaithfulAverage ctx precision: 0.25 | ctx recall: 0.75 | faithfulness: 0.72 | answer correct: 0.5
Questions 1 and 2 pass. Question 3 is a retrieval failure: the premium shipping chunk was never retrieved, so context recall is 0 and the answer invented a fee. Question 4 is a generation failure: the right chunk was retrieved, but the answer says 16 characters instead of 12, which lowers faithfulness. Averaging gives only a single number, whereas the per-question table shows where to work. The overlap-based faithfulness here is a crude proxy, and production systems use stronger checks.

Knowledge Check

Q1. An answer is faithful to its context but still wrong. What is the most likely cause?
Q2. Context recall is low for a question. Where should you look first?
Q3. Why must an LLM-as-a-judge be calibrated against human labels?

Key Points

✓
Evaluate retrieval and generation separately, then the whole system.
✓
Context precision and context recall describe retrieval; recall is the clearest sign that retrieval is the bottleneck.
✓
Generation has answer relevance, faithfulness (groundedness) and correctness (factuality), which can fail independently.
✓
Faithful does not mean correct: wrong sources produce faithful but wrong answers.
✓
A golden dataset of real, labeled questions is the ruler for every change; review synthetic questions.
✓
Use automated metrics for speed, calibrated LLM judges for scale and humans for trust and high-stakes cases.
✓
Failure analysis walks the pipeline from left to right and stops at the first broken stage; rerun the golden set after every change.

The Big Picture

Evaluation is what turns RAG from craft into engineering. With a golden set and a handful of metrics, every idea in this course becomes a testable hypothesis: change one thing, measure, keep it or drop it.
Golden questions → Run the system → Measure retrieval and generation → Diagnose → Improve → Repeat
Up Next: RAG Optimization and Performance. You will start the Production phase by tuning chunks, embeddings, retrieval, reranking and context, and by controlling latency, throughput, caching and cost.