Test Your LLM App with pytest: An Eval Harness in 40 Lines
Build a small LLM evaluation harness with pytest and a local model, then see why six out of six does not mean your prompt is good.
Video #06 ยท LLM engineering and evals
Steps in the video
Files
- cases.py (9 lines)
- compare_prompts.py (28 lines)
- judge.py (34 lines)
- test_triage.py (11 lines)
- tickets.py (24 lines)
Tested with
Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1, pytest==9.2.0
What this does not show
Six made-up emails, one small local model (llama3.2), one run at temperature 0, which hides run-to-run variation. The judge check uses one pair of summaries, so it shows order can decide the pick, not how often. Not a benchmark.
Sources
- https://dev.to/velsof/building-a-production-llm-evaluation-harness-in-pytest-cost-bounded-flake-aware-ci-gated-26pc
- https://futureagi.com/blog/evaluating-llm-judge-bias-mitigation-2026/
- https://machinelearningmastery.com/building-a-custom-llm-evaluation-harness-in-python-from-scratch/
- https://memx.app/blog/llm-as-a-judge-biases-and-fixes/
Learning IT by Example.