learning-it by example

#06 LLM engineering and evals 5.9 min

Test Your LLM App with pytest: An Eval Harness in 40 Lines

Build a small LLM evaluation harness with pytest and a local model, then see why six out of six does not mean your prompt is good.

1. What three parts make up a simple evaluation harness for an LLM app?
2. A careful prompt and the much shorter prompt 'Summarise the email.' were scored on the same six support emails. What did that show?
3. Why ask an LLM judge the same question twice with the two answers swapped?
4. Which check does the video recommend for the ticket category field?
5. How does the pytest file run one test for each golden email?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Write an awkward case first

    Copy the code folder first. Think of one support email that mixes two problems, for example a double charge that also crashes the app. Decide which category you think is right and write it down before you run anything. Add it to cases.py, then run python -m pytest -q test_triage.py. This needs Ollama with llama3.2 and takes a few minutes. Does the model agree with you?

    A hint

    Writing the expected answer first matters because afterwards it is too easy to agree with whatever the model said.

  2. Score a third prompt

    In compare_prompts.py add a third prompt to PROMPTS, such as 'You are a helpful assistant.' Predict its score out of 6 before you run python compare_prompts.py, and say why. The video found that a very short prompt also scored 6 of 6.

    A hint

    The schema already lists the four categories, so even a short prompt knows the choices.

  3. Check the judge with your own summary

    In judge.py write a third summary of the email that says the same thing as good in your own words. Add a line such as compare('my summary', good, mine). Predict whether the judge will be consistent when the order is swapped, then run python judge.py.

    A hint

    Consistent means the judge picks A then B, or B then A, so that it prefers the same text in both orders.

  4. Make a test fail on purpose

    In cases.py change the expected category of the dark mode email from feature to bug. Predict how many tests will pass and what pytest will show for the failing one. Then run python -m pytest -q test_triage.py and read the report. Change it back afterwards.

    A hint

    The video says pytest names the exact email that failed, so you know where to look.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six made-up emails, one small local model (llama3.2), one run at temperature 0, which hides run-to-run variation. The judge check uses one pair of summaries, so it shows order can decide the pick, not how often. Not a benchmark.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos