#304 Forward deployed engineer interviews 5.9 min
Scope an AI Pilot: Metric, Baseline and Stop Rule
Both pilots were faster. One still failed a guardrail that was written before the results.
Run the pilot judge
Copy the code folder first. Run python run.py. Find the baseline line and write down today's median minutes and wrong-team rate. Then check which pilot passes and which fails. You should see median 9.3 min and 7.5% for today, pilot A PASS and pilot B FAIL.
A hint
The last line of each pilot block ends with PASS or FAIL.
Tighten the success metric
In rule.py change the value of "faster" in RULE from 0.15 to 0.25. Predict the verdict for pilot A and pilot B, then run python run.py. With this change both pilots print FAIL, and pilot A fails the faster check. Why is it fair that you set this before looking at results?
A hint
Pilot A is 24% faster, just under the new limit.
Loosen the wrong-team guardrail
In rule.py change "wrong_extra" from 0.02 to 0.12. Predict what happens to pilot B and to its stop line, then run it. With this change pilot B prints PASS and the stop rule no longer fires. Write one sentence on why changing a rule after seeing the results is a mistake the video warns about.
A hint
The stop rule uses twice the wrong_extra value.
Make the helper too expensive
In tickets.py change the AI_COST for "A" from 0.28 to 1.50. Predict which check fails for pilot A, then run python run.py. Pilot A now shows cost 5.90, the cost check says no and the verdict is FAIL. Then try 1.20 and see where the cost check passes again.
A hint
Today's cost is 5.87 per ticket, and the cap is 100% of that.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Simulated tickets and pilots with rates and costs chosen by hand and one fixed seed; with other seeds the same pilot A fails about one time in three. The stop rule is printed, not enforced. No AI helper is run.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
Related videos
- #302 Forward Deployed Engineer Interview: What Is Tested Forward deployed engineer interviews
- #303 The FDE Decomposition Round: Vague Ask to a Plan Forward deployed engineer interviews
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys Forward deployed engineer interviews
- #306 Webhooks Done Right: Signatures, Retries and Replays Forward deployed engineer interviews
