learning-it by example

#43 RAG and vector search 5.7 min

Evaluate RAG: Recall, Faithfulness and a Wrong Checker

An early version of my RAG check passed the answer 'I don't know'. The fixed version still gets things wrong.

1. Why do you measure retrieval and answers separately when evaluating a RAG pipeline?
2. In the bike shop test, 500 character chunks found the answering phrase for how many of the 10 golden questions at k equals 5, and why not all 10?
3. In the bike shop RAG test with llama3.2 (one run), the worst ranked chunk and a loose prompt printed a refused count of 0/10 although 8 of the 10 answers said there was no information. Why?
4. In the bike shop RAG test with llama3.2 (one run), the model answered that brake pads are not typically covered by a standard vehicle warranty, using general knowledge rather than the handbook. Why did the faithful check pass it?
5. In an earlier llama3.2 run of the bike shop RAG test, the model said the shop closes at four on Saturday, which is right, yet the correct check marked it wrong. What is the reason?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Count the chunks

    Copy the code folder first. Open a Python prompt in the copy. Read handbook.md into a variable called text, then import headings, sentences and fixed from chunkers. Predict how many chunks come back from headings(text), sentences(text, 3) and fixed(text, 500). Then run them and compare with the table in the video (9, 13 and 7). Now try sentences(text, 2) and fixed(text, 250) and note how the counts change.

    A hint

    Use len() on each result. When I ran it, sentences(text, 2) gave 20 chunks and fixed(text, 250) gave 13.

  2. Grade a refusal

    In a Python prompt in the code folder, import correct from grade. Predict, then run correct("I don't know.", "about 35 miles in turbo mode") and correct("It is about 35 miles.", "about 35 miles in turbo mode"). Then try the answer "I don't know." against the phrase "brake pads and chains are wear items", which has no figures. Explain in one sentence why a refusal must not count as correct.

    A hint

    When I ran them, the first and third calls gave False and the second gave True.

  3. Reproduce the false miss

    Call correct("The shop closes at 4.", "Saturday from 10 to 4") and read the result. The answer is right, so find out why the check disagrees. Then change only the answer so that the call returns True without making the answer wrong. Write down what this tells you about checking figures only.

    A hint

    The check needs every figure of the answering phrase to be in the answer. Look at the numbers in the phrase.

  4. Fool the faithful check

    Import faithful from grade. Predict, then run faithful("Delivery costs 12 pounds", "Delivery costs 8 pounds") and faithful("Delivery is always free in the shop", "Delivery costs 8 pounds"). Then write your own made up answer that is clearly not supported by the context but still returns True. Say what a stronger check would have to do.

    A hint

    The second call returns True when run, because it has no figures. A claim level check, as the video suggests, would look at each claim.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One short made-up handbook, ten questions that I wrote, one small model (llama3.2) at temperature zero, one run. The two checks look only at figures and words; they are not the Ragas metric, and no claim-level or second-model check was tested. The worst chunk is an extreme case that I chose. Not a benchmark.

Up next in RAG and vector search

More in RAG and vector search

All RAG and vector search videos

Related videos