6 RAG Failure Modes Reproduced: Which Ones Actually Break
I tried to break a RAG app six ways. Four broke, one barely, and one did not break in five tries.
Video #47 ยท RAG and vector search
Steps in the video
Files
- answer_modes.py (26 lines)
- golden.py (13 lines)
- handbook.md (28 lines)
- more_modes.py (25 lines)
- rag.py (70 lines)
- retrieval_modes.py (16 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One short made-up handbook, three to ten questions per failure, one small model and one seeded run at temperature 0; the old copy is the handbook with 3 added to every figure; the model refused in five tries, which does not prove it never invents an answer.
Sources
Learning IT by Example.