learning-it by example

#68 LLM engineering and evals 5 min

Synthetic Test Cases: 27 Asked, 17 Passed Checks

I asked a small model for 27 test questions and only 17 passed three cheap checks.

1. A small model can write test questions for your LLM evals quickly. According to the video, where does the real work lie?
2. In the synthetic test cases video, a small model (llama3.2) was asked for 27 test questions from a nine-section handbook and wrote 42 lines. After the form check, the grounding check and the duplicate check, how many cases were kept?
3. In the video's synthetic test case filters, what does the grounding check ask about a generated answer?
4. In the video's code, when is a new generated question dropped as a duplicate?
5. In the video's test, a small model (llama3.2) wrote cases from a made-up handbook and 17 were kept after three cheap checks. What does the video say you still have to do with them?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Try the form check by hand

    Copy the code folder first. Open a Python prompt inside it and run: from filters import parse. Then call parse on these three lines: 'Q: What is the refund time? | A: 5 working days', 'Here are three test questions based on the passage:' and 'Q: What is the refund time? A: 5 working days'. Predict which return a question and answer pair and which return None, then run them. The first returns ('What is the refund time?', '5 working days') and the other two return None.

    A hint

    The pattern needs the letters Q:, then a bar, then A:. A line without the bar does not match.

  2. Is the answer in the passage?

    In the same Python prompt, load the Returns section: body = open('handbook.md', encoding='utf-8').read().split('## ')[1]. Now call grounded on this body with these answers: '30 DAYS!', 'within 30 days of delivery', 'about a month' and an empty string. Predict each result before you run it. You should get True, True, False and False. Then write one answer of your own that is true to the passage but fails the check, and say why it fails.

    A hint

    The check ignores capital letters and punctuation, but the words themselves must appear in the passage in the same order.

  3. Move the duplicate limit

    Call duplicate('How long do returns take to arrive?', ['How long do returns take?']) and predict the result, then run it. You should get True, because the two questions share 5 of 7 distinct words, which is about 0.71. Now edit filters.py and change 0.7 to 0.8, restart Python and run the same call. It should now give False. Explain in one sentence what the higher limit does to the number of questions kept. The docstring in filters.py still says 70 percent after your edit. When you have finished, change 0.8 back to 0.7.

    A hint

    Count the distinct words in each question. The check divides the words they share by all the words in the two together.

  4. Predict a stricter prompt

    This task needs Ollama with llama3.2, so treat it as a prediction first. In gen.py change the prompt from 'Write 3 test questions' to 'Write 1 test question'. Before you run python report.py, predict the first number on the printed line, the lines from the model, if the model follows the prompt exactly. Then run it and compare. Did any line of chat, such as an introduction, change the count?

    A hint

    There are 9 sections in handbook.md. Count how many lines you would expect per section, and remember that report.py counts every non-empty line.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A made-up nine-section handbook, one small local model (llama3.2), one run. The checks test form and whether the answer text appears in the passage, not whether the question is good or the answer is the right one. Only the first rejected line of each kind was read. The duplicate check caught nothing here, so it is untested. Read the kept cases yourself.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos