#47 RAG and vector search 5.9 min
6 RAG Failure Modes Reproduced: Which Ones Actually Break
I tried to break a RAG app six ways. Four broke, one barely, and one did not break in five tries.
Make an old copy
Copy the code folder first. Open a Python prompt in it and import old_version and TEXT from rag. Predict what old_version("A basic service costs 45 pounds and takes 2 days.") returns, then run it. Next print the first 150 characters of old_version(TEXT) and compare the Returns section with the original in handbook.md.
A hint
It adds 3 to every number in the text. When I ran the first call it gave 48 pounds and 5 days.
Sweep the chunk size
With the same Ollama setup as the video, write a loop over the sizes 60, 250, 500 and 1000. For each, cut the handbook with by_size, build an Index, and count how many of the 10 GOLDEN questions have the answering phrase in the nearest chunk, as retrieval_modes.py does. Predict whether the score rises or falls with size, then run it and note where it stops improving.
A hint
Use Index(chunks).top(question)[0] and the check needle in chunks[...]. My run gave 2, 8, 8 and 9 out of 10 for those sizes, with 120 characters giving 7.
Fool the refusal check
Import refused from rag. Predict, then run it on three answers: "NOT FOUND", "Refunds go back within 5 working days." and "It does not take long, about 5 days.". The third one answers the question. Explain why the check gets it wrong and how you would make a refusal check less fooled.
A hint
The check looks for phrases such as does not. When I ran it, the first and third gave True and the second gave False.
Add a question the handbook cannot answer
In answer_modes.py add a sixth question to MISSING that the handbook clearly does not answer, such as one about gift wrapping. Predict whether the model will say the answer is not there, then run python answer_modes.py if Ollama is installed and read the answer it gives. Compare it with your prediction, and note that one run does not prove the model never invents answers.
A hint
The model runs on your machine, so this may take a minute or more. Read the printed answer, do not only count refusals.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One short made-up handbook, three to ten questions per failure, one small model and one seeded run at temperature 0; the old copy is the handbook with 3 added to every figure; the model refused in five tries, which does not prove it never invents an answer.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #52 RAG Citations: Source Removed, Exact Quote, Wrong Source RAG and vector search
- #26 Plan, Act, Reflect: Does It Help a Small Model? AI agents and MCP
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama) LLM engineering and evals
