Synthetic Test Cases: 27 Asked, 17 Passed Checks
I asked a small model for 27 test questions and only 17 passed three cheap checks.
Video #68 ยท LLM engineering and evals
Steps in the video
Files
- filters.py (27 lines)
- gen.py (14 lines)
- handbook.md (28 lines)
- report.py (29 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
A made-up nine-section handbook, one small local model (llama3.2), one run. The checks test form and whether the answer text appears in the passage, not whether the question is good or the answer is the right one. Only the first rejected line of each kind was read. The duplicate check caught nothing here, so it is untested. Read the kept cases yourself.
Sources
Learning IT by Example.