learning-it by example

Back to the video

6 RAG Failure Modes Reproduced: Which Ones Actually Break

I tried to break a RAG app six ways. Four broke, one barely, and one did not break in five tries.

Video #47 ยท RAG and vector search

Steps in the video

  1. 0:00 Six failures
  2. 0:24 The idea
  3. 0:54 Setup
  4. 1:13 Two helpers
  5. 1:26 Failures one and two
  6. 2:19 Failures three and four
  7. 3:18 Failures five and six
  8. 4:02 What this does not show
  9. 4:33 Mistakes to avoid
  10. 4:56 My take and next steps
  11. 5:22 Recap

Files

  • answer_modes.py (26 lines)
  • golden.py (13 lines)
  • handbook.md (28 lines)
  • more_modes.py (25 lines)
  • rag.py (70 lines)
  • retrieval_modes.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One short made-up handbook, three to ten questions per failure, one small model and one seeded run at temperature 0; the old copy is the handbook with 3 added to every figure; the model refused in five tries, which does not prove it never invents an answer.

Sources

Learning IT by Example.