#317 Forward deployed engineer interviews 6 min
Is Prompt v2 Really Better? Evals With Small Samples
Version 2 scored 80% against 70%. On 40 cases, I still could not tell them apart.
Make the two versions identical
Copy the code folder first. Open cases.py and change PATTERN to [((1, 1), 28), ((0, 0), 12)], so that both versions are right on the same 28 cases and wrong on the same 12. Predict the gap and the verdict for 40 cases, then run python run.py and compare.
A hint
Look at the line that starts with 'paired gap'. With no case where the versions differ, every difference is zero.
Give version 2 a bigger lead
In cases.py change PATTERN to [((1, 1), 20), ((0, 1), 12), ((1, 0), 3), ((0, 0), 5)]. It is still 40 cases. Before you run it, predict whether the verdict for 40 cases will still be 'cannot tell them apart'. Then run python run.py and compare your guess with the printed gap and interval. Then put PATTERN back to [((1, 1), 25), ((0, 1), 7), ((1, 0), 3), ((0, 0), 5)], because the next exercises need the original.
A hint
When I ran it, version 1 scored 23/40 and the 40-case gap was +22%, with an interval of +5% to +40%.
Find how many cases are enough
In run.py change the line 'for copies in (1, 10):' to 'for copies in (1, 2, 3, 4, 10):'. Predict how many cases are needed before the interval for the paired gap is entirely above zero. Run python run.py and read the 'paired gap' lines to check.
A hint
Each extra copy repeats the same 40-case pattern, so the gap stays at +10% and only the margin changes. I found the verdict changed at 120 cases.
Change the resampling settings
In stats.py, in the paired_gap function, change reps=10000 to reps=200 and seed=1 to seed=5. Predict which printed numbers will move and which will stay the same, then run python run.py. Compare the 'paired gap' lines with the 'formula agrees' lines.
A hint
I saw the 40-case interval become -2% to +28% and the 400-case interval become +4% to +17%, while the formula lines stayed the same.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Forty made-up cases; the 400 are the same 40 repeated ten times, so they show how the margin shrinks, not what a bigger set would find. Cases are treated as an independent random sample. 95 percent is a convention. Not proven is not the same as no difference. Only the paper's summary was read.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #56 Prompt Versioning: Compare Changes Case by Case LLM engineering and evals
- #133 A/B Test Analysis in SQL and Python: z Test and Peeking SQL and analytics
- #65 A pytest Eval Gate: Fail the Build When Pass Rate Drops LLM engineering and evals
- #302 Forward Deployed Engineer Interview: What Is Tested Forward deployed engineer interviews
