learning-it by example

#317 Forward deployed engineer interviews 6 min

Is Prompt v2 Really Better? Evals With Small Samples

Version 2 scored 80% against 70%. On 40 cases, I still could not tell them apart.

1. Version 2 of a prompt scored 80% and version 1 scored 70% on 40 test cases. What did the video conclude?
2. Why does the video compare two prompt versions with a paired gap rather than by looking at each version's own margin?
3. The 40-case pattern was repeated ten times to give 400 cases. What did the paired gap interval show, and what was the verdict?
4. In the prompt version 1 versus version 2 error-bars demo, what does the video say about the 400-case run?
5. In the eval script that compares prompt version 1 and version 2 with error bars, when does the program print 'version 2 is better'?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Make the two versions identical

    Copy the code folder first. Open cases.py and change PATTERN to [((1, 1), 28), ((0, 0), 12)], so that both versions are right on the same 28 cases and wrong on the same 12. Predict the gap and the verdict for 40 cases, then run python run.py and compare.

    A hint

    Look at the line that starts with 'paired gap'. With no case where the versions differ, every difference is zero.

  2. Give version 2 a bigger lead

    In cases.py change PATTERN to [((1, 1), 20), ((0, 1), 12), ((1, 0), 3), ((0, 0), 5)]. It is still 40 cases. Before you run it, predict whether the verdict for 40 cases will still be 'cannot tell them apart'. Then run python run.py and compare your guess with the printed gap and interval. Then put PATTERN back to [((1, 1), 25), ((0, 1), 7), ((1, 0), 3), ((0, 0), 5)], because the next exercises need the original.

    A hint

    When I ran it, version 1 scored 23/40 and the 40-case gap was +22%, with an interval of +5% to +40%.

  3. Find how many cases are enough

    In run.py change the line 'for copies in (1, 10):' to 'for copies in (1, 2, 3, 4, 10):'. Predict how many cases are needed before the interval for the paired gap is entirely above zero. Run python run.py and read the 'paired gap' lines to check.

    A hint

    Each extra copy repeats the same 40-case pattern, so the gap stays at +10% and only the margin changes. I found the verdict changed at 120 cases.

  4. Change the resampling settings

    In stats.py, in the paired_gap function, change reps=10000 to reps=200 and seed=1 to seed=5. Predict which printed numbers will move and which will stay the same, then run python run.py. Compare the 'paired gap' lines with the 'formula agrees' lines.

    A hint

    I saw the 40-case interval become -2% to +28% and the 400-case interval become +4% to +17%, while the formula lines stayed the same.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Forty made-up cases; the 400 are the same 40 repeated ten times, so they show how the margin shrinks, not what a bigger set would find. Cases are treated as an independent random sample. 95 percent is a convention. Not proven is not the same as no difference. Only the paper's summary was read.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos