learning-it by example

Back to the video

Is Prompt v2 Really Better? Evals With Small Samples

Version 2 scored 80% against 70%. On 40 cases, I still could not tell them apart.

Video #317 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 Version 2 scored 80 percent, version 1 scored 70, and I could not tell them apart
  2. 0:32 The idea
  3. 1:06 Setup
  4. 1:33 The same cases, two versions
  5. 2:02 An interval for a score
  6. 2:21 An interval for the gap
  7. 2:42 The same, by formula
  8. 3:02 Ask twice
  9. 4:00 What this does not show
  10. 4:36 Mistakes to avoid
  11. 5:02 My take and next steps
  12. 5:28 Recap

Files

  • cases.py (12 lines)
  • run.py (16 lines)
  • stats.py (32 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Forty made-up cases; the 400 are the same 40 repeated ten times, so they show how the margin shrinks, not what a bigger set would find. Cases are treated as an independent random sample. 95 percent is a convention. Not proven is not the same as no difference. Only the paper's summary was read.

Sources

Learning IT by Example.