learning-it by example

Back to the video

Scope an AI Pilot: Metric, Baseline and Stop Rule

Both pilots were faster. One still failed a guardrail that was written before the results.

Video #304 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 This pilot was faster, and it still failed
  2. 0:41 The idea
  3. 1:16 Setup
  4. 1:44 Today, measured first
  5. 2:09 The rule, written first
  6. 2:40 The verdict and the stop rule
  7. 3:03 Judge both pilots
  8. 3:51 What this does not show
  9. 4:31 Mistakes to avoid
  10. 4:56 My take and next steps
  11. 5:23 Recap

Files

  • rule.py (35 lines)
  • run.py (18 lines)
  • tickets.py (23 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Simulated tickets and pilots with rates and costs chosen by hand and one fixed seed; with other seeds the same pilot A fails about one time in three. The stop rule is printed, not enforced. No AI helper is run.

Sources

Learning IT by Example.