Evaluate RAG: Recall, Faithfulness and a Wrong Checker
An early version of my RAG check passed the answer 'I don't know'. The fixed version still gets things wrong.
Video #43 ยท RAG and vector search
Steps in the video
Files
- chunkers.py (17 lines)
- generate.py (44 lines)
- golden.py (13 lines)
- grade.py (22 lines)
- handbook.md (28 lines)
- retrieval.py (42 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One short made-up handbook, ten questions that I wrote, one small model (llama3.2) at temperature zero, one run. The two checks look only at figures and words; they are not the Ragas metric, and no claim-level or second-model check was tested. The worst chunk is an extreme case that I chose. Not a benchmark.
Sources
Learning IT by Example.