learning-it by example

#65 LLM engineering and evals 6.4 min

A pytest Eval Gate: Fail the Build When Pass Rate Drops

A made-up change dropped the pass rate from 90 to 60 percent, and the pytest gate exited 1.

1. What is an eval gate in the pytest setup shown in the video?
2. In the video's pytest eval gate run, what pass rate and exit code did version 1 and version 2 of the stand-in assistant get?
3. How does a failing pytest eval gate stop a build in a GitHub Actions workflow?
4. In the video's pytest eval gate, how does the check function decide that an answer passes a case?
5. What limits does the video give for this phrase-checking eval gate?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the gate on both versions

    Copy the code folder first and install the pinned pytest if you do not have it. Predict the pass rate and exit code for each version, then run python gate.py v1 and python gate.py v2. You should see 90 percent and exit code 0 for v1, and 60 percent with exit code 1 for v2, with refund, contact, plan and medical failing.

    A hint

    The assistant is a dictionary of fixed answers, so the output repeats on every run apart from the timing.

  2. Fix the known weak answer

    In assistant.py change the V1 contact answer to 'Email help@example.com or use the chat button.' Predict the new pass rates, then run python gate.py v1 and python gate.py v2. You should see v1 at 100 percent with none failed, and v2 at 70 percent with refund, plan and medical failing, so the gate still fails. Then change the contact answer back to 'Use the chat button in the app.', because the next exercises assume v1 at 90 percent and v2 at 60 percent.

    A hint

    The contact case requires the phrase example.com.

  3. Move the threshold

    In test_evals.py change THRESHOLD from 0.9 to 1.0 and run python gate.py v1. Then change it to 0.5 and run python gate.py v2. Predict both results first. With 1.0, v1 fails with exit code 1 because 90 percent is below 100. With 0.5, v2 passes with exit code 0 at 60 percent. Write one sentence on why neither threshold is a good choice. Then change THRESHOLD back to 0.9.

    A hint

    The video warned against a threshold of 100 percent and against one so low that it never fails.

  4. Find a hole in the phrase check

    In assistant.py change the V1 plan answer from 'up to 10 users' to 'up to 110 users' and run python gate.py v1. Predict whether plan will pass, then look at the result: it still passes, because the phrase '10 users' appears inside '110 users'. Now change it to 'up to 100 users' and run again, and you should see plan fail. Write one sentence on what this shows about phrase checks. Then put the plan answer back to 'up to 10 users'.

    A hint

    The video said phrase checks only see words and can pass wrong answers that use the right words.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No model is called (the assistant is a dict of fixed, made-up answers), the ten cases are made up, and the GitHub Actions workflow is shown as text and was not run on GitHub; the pytest gate was run locally.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos