learning-it by example

Back to the video

Synthetic Test Cases: 27 Asked, 17 Passed Checks

I asked a small model for 27 test questions and only 17 passed three cheap checks.

Video #68 ยท LLM engineering and evals

Steps in the video

  1. 0:00 27 asked, 17 passed the checks
  2. 0:22 The idea
  3. 1:05 Setup
  4. 1:23 Asking for cases
  5. 1:39 Two checks
  6. 2:02 The third check
  7. 2:18 Count what survives
  8. 3:17 What this does not show
  9. 3:47 Mistakes to avoid
  10. 4:07 My take and next steps
  11. 4:38 Recap

Files

  • filters.py (27 lines)
  • gen.py (14 lines)
  • handbook.md (28 lines)
  • report.py (29 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

A made-up nine-section handbook, one small local model (llama3.2), one run. The checks test form and whether the answer text appears in the passage, not whether the question is good or the answer is the right one. Only the first rejected line of each kind was read. The duplicate check caught nothing here, so it is untested. Read the kept cases yourself.

Sources

Learning IT by Example.