Is Prompt v2 Really Better? Evals With Small Samples
Version 2 scored 80% against 70%. On 40 cases, I still could not tell them apart.
Video #317 ยท Forward deployed engineer interviews
Steps in the video
- 0:00 Version 2 scored 80 percent, version 1 scored 70, and I could not tell them apart
- 0:32 The idea
- 1:06 Setup
- 1:33 The same cases, two versions
- 2:02 An interval for a score
- 2:21 An interval for the gap
- 2:42 The same, by formula
- 3:02 Ask twice
- 4:00 What this does not show
- 4:36 Mistakes to avoid
- 5:02 My take and next steps
- 5:28 Recap
Files
- cases.py (12 lines)
- run.py (16 lines)
- stats.py (32 lines)
Tested with
Python 3.12.15
What this does not show
Forty made-up cases; the 400 are the same 40 repeated ten times, so they show how the margin shrinks, not what a bigger set would find. Cases are treated as an independent random sample. 95 percent is a convention. Not proven is not the same as no difference. Only the paper's summary was read.
Sources
Learning IT by Example.