learning-it by example

Back to the video

Evaluate RAG: Recall, Faithfulness and a Wrong Checker

An early version of my RAG check passed the answer 'I don't know'. The fixed version still gets things wrong.

Video #43 ยท RAG and vector search

Steps in the video

  1. 0:00 Grade the grader
  2. 0:24 The idea
  3. 0:49 Setup
  4. 1:06 The question set
  5. 1:21 Recall at k
  6. 2:02 Two checks in code
  7. 2:25 The three setups
  8. 3:19 What the checks missed
  9. 4:04 What this does not show
  10. 4:33 Mistakes to avoid
  11. 4:55 My take and next steps
  12. 5:20 Recap

Files

  • chunkers.py (17 lines)
  • generate.py (44 lines)
  • golden.py (13 lines)
  • grade.py (22 lines)
  • handbook.md (28 lines)
  • retrieval.py (42 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One short made-up handbook, ten questions that I wrote, one small model (llama3.2) at temperature zero, one run. The two checks look only at figures and words; they are not the Ragas metric, and no claim-level or second-model check was tested. The worst chunk is an extreme case that I chose. Not a benchmark.

Sources

Learning IT by Example.