#43 RAG and vector search 5.7 min
Evaluate RAG: Recall, Faithfulness and a Wrong Checker
An early version of my RAG check passed the answer 'I don't know'. The fixed version still gets things wrong.
Count the chunks
Copy the code folder first. Open a Python prompt in the copy. Read handbook.md into a variable called text, then import headings, sentences and fixed from chunkers. Predict how many chunks come back from headings(text), sentences(text, 3) and fixed(text, 500). Then run them and compare with the table in the video (9, 13 and 7). Now try sentences(text, 2) and fixed(text, 250) and note how the counts change.
A hint
Use len() on each result. When I ran it, sentences(text, 2) gave 20 chunks and fixed(text, 250) gave 13.
Grade a refusal
In a Python prompt in the code folder, import correct from grade. Predict, then run correct("I don't know.", "about 35 miles in turbo mode") and correct("It is about 35 miles.", "about 35 miles in turbo mode"). Then try the answer "I don't know." against the phrase "brake pads and chains are wear items", which has no figures. Explain in one sentence why a refusal must not count as correct.
A hint
When I ran them, the first and third calls gave False and the second gave True.
Reproduce the false miss
Call correct("The shop closes at 4.", "Saturday from 10 to 4") and read the result. The answer is right, so find out why the check disagrees. Then change only the answer so that the call returns True without making the answer wrong. Write down what this tells you about checking figures only.
A hint
The check needs every figure of the answering phrase to be in the answer. Look at the numbers in the phrase.
Fool the faithful check
Import faithful from grade. Predict, then run faithful("Delivery costs 12 pounds", "Delivery costs 8 pounds") and faithful("Delivery is always free in the shop", "Delivery costs 8 pounds"). Then write your own made up answer that is clearly not supported by the context but still returns True. Say what a stronger check would have to do.
A hint
The second call returns True when run, because it has no figures. A claim level check, as the video suggests, would look at each claim.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One short made-up handbook, ten questions that I wrote, one small model (llama3.2) at temperature zero, one run. The two checks look only at figures and words; they are not the Ragas metric, and no claim-level or second-model check was tested. The worst chunk is an extreme case that I chose. Not a benchmark.
Up next in RAG and vector search
More in RAG and vector search
Related videos
- #68 Synthetic Test Cases: 27 Asked, 17 Passed Checks LLM engineering and evals
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
