#52 RAG and vector search 5.7 min
RAG Citations: Source Removed, Exact Quote, Wrong Source
Every citation was real and every quote exact. With the right source removed, two still pointed at the wrong source.
Test the quote check
Copy the code folder first. Open a Python prompt in it and import verbatim and CHUNKS from rag. Predict, then run verbatim("Customers can return any bike within 30 days of delivery", 1), the same call with 2 as the section number, and verbatim("CUSTOMERS CAN RETURN any bike within 30 days", 1). Then change 30 to 31 in the quote and call it again. Which of the four calls print True and why?
A hint
Importing rag needs Ollama running with all-minilm, as in the video. When I ran it, the first and third calls gave True and the second and the 31 days quote gave False.
Pull ids out of an answer
Import cited_ids and quote_of from rag. Make up an answer in the form the prompt asks for, with ANSWER, SOURCE: [1] and QUOTE with the words in double quotes. Predict what cited_ids and quote_of return, then run them. Try the string NOT FOUND and the string SOURCE: [1] and [3]. Say what a citation checker should do when no id comes back.
A hint
When I ran it, cited_ids gave [1, 3] for the last string and an empty list for NOT FOUND.
Drop the right section
Use retrieve from rag for the question How many days do I have to return a bike? first with no drop, then with drop=1, which is the Returns section, as removed.py does. Predict which three section numbers come back each time, then run it. Look at the CHUNKS headings to see which sections they are. Why can a model be tempted to cite one of these sections when the right one is missing?
A hint
Importing rag needs Ollama with all-minilm. My run gave [1, 3, 2] without the drop and [3, 2, 7] with it.
Add a question the handbook cannot answer
Add a third question with None as its section to QUESTIONS in golden.py, for example one about gift wrapping. Predict how many of the 3 unanswerable questions check.py will count as refused and as cited anyway. The printed line still ends with the fixed text (of 2), because that label is typed into check.py, so read it as out of 3. If you can run python check.py, which takes several minutes without a graphics card, read the printed line and compare. One question is a very small test, so say what else you would try. Then remove your extra question again, so golden.py has the original twelve.
A hint
In the video both unanswerable questions were refused with no citation, so refused was 2 of 2 and cited anyway was 0 of 2.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One short made-up handbook and 12 questions I wrote, one small model (llama3.2), one run at temperature 0. Two wrong citations out of ten is a small count; the answer text was not checked against the quote.
Up next in RAG and vector search
More in RAG and vector search
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local)
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results
- #40 Chunking Strategies Compared on One Document
- #41 Hybrid Search: BM25 Plus Vectors, Tested
- #42 Reranking With a Cross-Encoder: Does It Help? Python Test
- #43 Evaluate RAG: Recall, Faithfulness and a Wrong Checker
Related videos
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
- #47 6 RAG Failure Modes Reproduced: Which Ones Actually Break RAG and vector search
- #26 Plan, Act, Reflect: Does It Help a Small Model? AI agents and MCP
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
