A pytest Eval Gate: Fail the Build When Pass Rate Drops
A made-up change dropped the pass rate from 90 to 60 percent, and the pytest gate exited 1.
Video #65 ยท LLM engineering and evals
Steps in the video
Files
- assistant.py (22 lines)
- cases.py (21 lines)
- conftest.py (10 lines)
- gate.py (9 lines)
- pytest.ini (2 lines)
- test_evals.py (13 lines)
Tested with
Python 3.12.15 | pytest==9.1.1
What this does not show
No model is called (the assistant is a dict of fixed, made-up answers), the ten cases are made up, and the GitHub Actions workflow is shown as text and was not run on GitHub; the pytest gate was run locally.
Sources
Learning IT by Example.