#65 LLM engineering and evals 6.4 min
A pytest Eval Gate: Fail the Build When Pass Rate Drops
A made-up change dropped the pass rate from 90 to 60 percent, and the pytest gate exited 1.
Run the gate on both versions
Copy the code folder first and install the pinned pytest if you do not have it. Predict the pass rate and exit code for each version, then run python gate.py v1 and python gate.py v2. You should see 90 percent and exit code 0 for v1, and 60 percent with exit code 1 for v2, with refund, contact, plan and medical failing.
A hint
The assistant is a dictionary of fixed answers, so the output repeats on every run apart from the timing.
Fix the known weak answer
In assistant.py change the V1 contact answer to 'Email help@example.com or use the chat button.' Predict the new pass rates, then run python gate.py v1 and python gate.py v2. You should see v1 at 100 percent with none failed, and v2 at 70 percent with refund, plan and medical failing, so the gate still fails. Then change the contact answer back to 'Use the chat button in the app.', because the next exercises assume v1 at 90 percent and v2 at 60 percent.
A hint
The contact case requires the phrase example.com.
Move the threshold
In test_evals.py change THRESHOLD from 0.9 to 1.0 and run python gate.py v1. Then change it to 0.5 and run python gate.py v2. Predict both results first. With 1.0, v1 fails with exit code 1 because 90 percent is below 100. With 0.5, v2 passes with exit code 0 at 60 percent. Write one sentence on why neither threshold is a good choice. Then change THRESHOLD back to 0.9.
A hint
The video warned against a threshold of 100 percent and against one so low that it never fails.
Find a hole in the phrase check
In assistant.py change the V1 plan answer from 'up to 10 users' to 'up to 110 users' and run python gate.py v1. Predict whether plan will pass, then look at the result: it still passes, because the phrase '10 users' appears inside '110 users'. Now change it to 'up to 100 users' and run again, and you should see plan fail. Write one sentence on what this shows about phrase checks. Then put the plan answer back to 'up to 10 users'.
A hint
The video said phrase checks only see words and can pass wrong answers that use the right words.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
No model is called (the assistant is a dict of fixed, made-up answers), the ten cases are made up, and the GitHub Actions workflow is shown as text and was not run on GitHub; the pytest gate was run locally.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #146 Test Data Code with pytest: Fixtures and Frames Python for data
- #20 AI Coding Assistant Workflow: Copy, Test, Read the Diff AI coding and productivity
- #317 Is Prompt v2 Really Better? Evals With Small Samples Forward deployed engineer interviews
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals