Few-Shot Examples: How Many Is Enough? A Measured Test
Zero examples scored 12 of 20. Eight scored 18. Flipped labels dropped it to 13.
Video #63 ยท LLM engineering and evals
Steps in the video
Files
- classify.py (29 lines)
- examples.py (34 lines)
- labels.py (8 lines)
- order.py (14 lines)
- sweep.py (8 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Twenty made-up test messages and eight made-up examples, all written by the author in one style, one made-up rule, one small local model (llama3.2), one run at temperature 0. A gap of one or two answers is within noise. Order was tested with three orders of one set of four examples. The label test flipped every label, which is not the random replacement tested in the cited paper. Not a benchmark.
Sources
Learning IT by Example.