learning-it by example

Back to the video

RAG Citations: Source Removed, Exact Quote, Wrong Source

Every citation was real and every quote exact. With the right source removed, two still pointed at the wrong source.

Video #52 ยท RAG and vector search

Steps in the video

  1. 0:00 Real and wrong
  2. 0:35 The idea
  3. 1:15 Setup
  4. 1:40 Numbered sources
  5. 2:00 Asking for a quote
  6. 2:23 Checking the quote
  7. 3:20 Two wrong citations
  8. 3:50 What this does not show
  9. 4:21 Mistakes to avoid
  10. 4:45 My take and next steps
  11. 5:12 Recap

Files

  • check.py (26 lines)
  • golden.py (15 lines)
  • handbook.md (28 lines)
  • rag.py (60 lines)
  • removed.py (24 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One short made-up handbook and 12 questions I wrote, one small model (llama3.2), one run at temperature 0. Two wrong citations out of ten is a small count; the answer text was not checked against the quote.

Sources

Learning IT by Example.