learning-it by example

Back to the video

Test Your LLM App with pytest: An Eval Harness in 40 Lines

Build a small LLM evaluation harness with pytest and a local model, then see why six out of six does not mean your prompt is good.

Video #06 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Test your LLM app
  2. 0:21 What a harness is
  3. 0:46 Setup
  4. 1:01 The app under test
  5. 1:20 The golden set
  6. 1:47 One test per case
  7. 2:22 Mistake one
  8. 3:15 Mistake two
  9. 4:07 What this does not show
  10. 4:40 Mistakes to avoid
  11. 5:07 My take and next steps
  12. 5:37 Recap

Files

  • cases.py (9 lines)
  • compare_prompts.py (28 lines)
  • judge.py (34 lines)
  • test_triage.py (11 lines)
  • tickets.py (24 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1, pytest==9.2.0

What this does not show

Six made-up emails, one small local model (llama3.2), one run at temperature 0, which hides run-to-run variation. The judge check uses one pair of summaries, so it shows order can decide the pick, not how often. Not a benchmark.

Sources

Learning IT by Example.