learning-it by example

Back to the video

A pytest Eval Gate: Fail the Build When Pass Rate Drops

A made-up change dropped the pass rate from 90 to 60 percent, and the pytest gate exited 1.

Video #65 ยท LLM engineering and evals

Steps in the video

  1. 0:00 A made-up change dropped the pass rate from 90 to 60 percent. The gate said no.
  2. 0:34 The idea
  3. 1:04 Setup
  4. 1:26 The stand-in assistant
  5. 1:59 The cases
  6. 2:25 The check
  7. 2:39 The gate
  8. 3:02 Run it like CI
  9. 4:00 The workflow
  10. 4:33 What this does not show
  11. 4:59 Mistakes to avoid
  12. 5:27 My take and next steps
  13. 5:57 Recap

Files

  • assistant.py (22 lines)
  • cases.py (21 lines)
  • conftest.py (10 lines)
  • gate.py (9 lines)
  • pytest.ini (2 lines)
  • test_evals.py (13 lines)

Download the code (zip)

Tested with

Python 3.12.15 | pytest==9.1.1

What this does not show

No model is called (the assistant is a dict of fixed, made-up answers), the ten cases are made up, and the GitHub Actions workflow is shown as text and was not run on GitHub; the pytest gate was run locally.

Sources

Learning IT by Example.