RAG Citations: Source Removed, Exact Quote, Wrong Source
Every citation was real and every quote exact. With the right source removed, two still pointed at the wrong source.
Video #52 ยท RAG and vector search
Steps in the video
Files
- check.py (26 lines)
- golden.py (15 lines)
- handbook.md (28 lines)
- rag.py (60 lines)
- removed.py (24 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One short made-up handbook and 12 questions I wrote, one small model (llama3.2), one run at temperature 0. Two wrong citations out of ten is a small count; the answer text was not checked against the quote.
Sources
Learning IT by Example.