#68 LLM engineering and evals 5 min
Synthetic Test Cases: 27 Asked, 17 Passed Checks
I asked a small model for 27 test questions and only 17 passed three cheap checks.
Try the form check by hand
Copy the code folder first. Open a Python prompt inside it and run: from filters import parse. Then call parse on these three lines: 'Q: What is the refund time? | A: 5 working days', 'Here are three test questions based on the passage:' and 'Q: What is the refund time? A: 5 working days'. Predict which return a question and answer pair and which return None, then run them. The first returns ('What is the refund time?', '5 working days') and the other two return None.
A hint
The pattern needs the letters Q:, then a bar, then A:. A line without the bar does not match.
Is the answer in the passage?
In the same Python prompt, load the Returns section: body = open('handbook.md', encoding='utf-8').read().split('## ')[1]. Now call grounded on this body with these answers: '30 DAYS!', 'within 30 days of delivery', 'about a month' and an empty string. Predict each result before you run it. You should get True, True, False and False. Then write one answer of your own that is true to the passage but fails the check, and say why it fails.
A hint
The check ignores capital letters and punctuation, but the words themselves must appear in the passage in the same order.
Move the duplicate limit
Call duplicate('How long do returns take to arrive?', ['How long do returns take?']) and predict the result, then run it. You should get True, because the two questions share 5 of 7 distinct words, which is about 0.71. Now edit filters.py and change 0.7 to 0.8, restart Python and run the same call. It should now give False. Explain in one sentence what the higher limit does to the number of questions kept. The docstring in filters.py still says 70 percent after your edit. When you have finished, change 0.8 back to 0.7.
A hint
Count the distinct words in each question. The check divides the words they share by all the words in the two together.
Predict a stricter prompt
This task needs Ollama with llama3.2, so treat it as a prediction first. In gen.py change the prompt from 'Write 3 test questions' to 'Write 1 test question'. Before you run python report.py, predict the first number on the printed line, the lines from the model, if the model follows the prompt exactly. Then run it and compare. Did any line of chat, such as an introduction, change the count?
A hint
There are 9 sections in handbook.md. Count how many lines you would expect per section, and remember that report.py counts every non-empty line.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A made-up nine-section handbook, one small local model (llama3.2), one run. The checks test form and whether the answer text appears in the passage, not whether the question is good or the answer is the right one. Only the first rejected line of each kind was read. The duplicate check caught nothing here, so it is untested. Read the kept cases yourself.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
- #64 Chain of Thought: When Step by Step Helps, and Its Cost LLM engineering and evals
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
