#06 LLM engineering and evals 5.9 min
Test Your LLM App with pytest: An Eval Harness in 40 Lines
Build a small LLM evaluation harness with pytest and a local model, then see why six out of six does not mean your prompt is good.
Write an awkward case first
Copy the code folder first. Think of one support email that mixes two problems, for example a double charge that also crashes the app. Decide which category you think is right and write it down before you run anything. Add it to cases.py, then run python -m pytest -q test_triage.py. This needs Ollama with llama3.2 and takes a few minutes. Does the model agree with you?
A hint
Writing the expected answer first matters because afterwards it is too easy to agree with whatever the model said.
Score a third prompt
In compare_prompts.py add a third prompt to PROMPTS, such as 'You are a helpful assistant.' Predict its score out of 6 before you run python compare_prompts.py, and say why. The video found that a very short prompt also scored 6 of 6.
A hint
The schema already lists the four categories, so even a short prompt knows the choices.
Check the judge with your own summary
In judge.py write a third summary of the email that says the same thing as good in your own words. Add a line such as compare('my summary', good, mine). Predict whether the judge will be consistent when the order is swapped, then run python judge.py.
A hint
Consistent means the judge picks A then B, or B then A, so that it prefers the same text in both orders.
Make a test fail on purpose
In cases.py change the expected category of the dark mode email from feature to bug. Predict how many tests will pass and what pytest will show for the failing one. Then run python -m pytest -q test_triage.py and read the report. Change it back afterwards.
A hint
The video says pytest names the exact email that failed, so you know where to look.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Six made-up emails, one small local model (llama3.2), one run at temperature 0, which hides run-to-run variation. The judge check uses one pair of summaries, so it shows order can decide the pick, not how often. Not a benchmark.
Up next in LLM engineering and evals
More in LLM engineering and evals
Related videos
- #56 Prompt Versioning: Compare Changes Case by Case LLM engineering and evals
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #12 Multi-Agent Patterns in Plain Python: Pipeline, Router, Loop AI agents and MCP
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
