learning-it by example

Back to the video

Few-Shot Examples: How Many Is Enough? A Measured Test

Zero examples scored 12 of 20. Eight scored 18. Flipped labels dropped it to 13.

Video #63 ยท LLM engineering and evals

Steps in the video

  1. 0:00 How many examples?
  2. 0:22 The idea
  3. 0:52 Setup
  4. 1:05 The task and the rule
  5. 1:28 The twenty test messages
  6. 1:46 The prompt
  7. 2:34 Order and the misses
  8. 3:52 What this does not show
  9. 4:20 Mistakes to avoid
  10. 4:45 My take and next steps
  11. 5:13 Recap

Files

  • classify.py (29 lines)
  • examples.py (34 lines)
  • labels.py (8 lines)
  • order.py (14 lines)
  • sweep.py (8 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Twenty made-up test messages and eight made-up examples, all written by the author in one style, one made-up rule, one small local model (llama3.2), one run at temperature 0. A gap of one or two answers is within noise. Order was tested with three orders of one set of four examples. The label test flipped every label, which is not the random replacement tested in the cited paper. Not a benchmark.

Sources

Learning IT by Example.