#63 LLM engineering and evals 5.6 min
Few-Shot Examples: How Many Is Enough? A Measured Test
Zero examples scored 12 of 20. Eight scored 18. Flipped labels dropped it to 13.
Read the house rule from the examples
Copy the code folder first. Open examples.py and read the eight EXAMPLES. In your own words, write the house rule that the labels show, and say which of the eight examples show that anger is not urgent. The prompt never states the rule, so only the examples teach it.
A hint
The video said money taken wrongly, account security and safety are urgent, and anger is not.
Print the prompts and read them
Run python -c with this idea, or add a small script: import build from classify and EXAMPLES from examples, then print build('Hello there', EXAMPLES[:2]) and print(build('Hello there', [])). The first prompt should hold the instruction, two examples that each end in a label, then 'Message: Hello there' and 'Label:'. The second should hold only the instruction and the new message. Check that this is what you see.
A hint
You need the ollama package installed for the import to work, but this does not call the model.
Predict a score with your own examples
Write down which two of the eight examples you would choose if you could only use two, and predict whether the score will beat the 16 of 20 that the first two examples got in the video. If Ollama and llama3.2 are installed, write a small script that imports accuracy from classify and TESTS and EXAMPLES from examples, and call accuracy([EXAMPLES[i], EXAMPLES[j]], TESTS) for your two examples.
A hint
Keep an urgent example and a normal example, so the pair stays balanced.
Add a trap message of your own
Add a twenty-first tuple to TESTS in examples.py, a message that sounds urgent but is normal by the house rule, such as a very angry message about a missing leaflet. Predict whether the 8-example prompt will get it right. If you can run Ollama, run python order.py and look at whether your message appears in the misses list.
A hint
The video said the traps are what you need to check, not only the easy cases.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Twenty made-up test messages and eight made-up examples, all written by the author in one style, one made-up rule, one small local model (llama3.2), one run at temperature 0. A gap of one or two answers is within noise. Order was tested with three orders of one set of four examples. The label test flipped every label, which is not the random replacement tested in the cited paper. Not a benchmark.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #64 Chain of Thought: When Step by Step Helps, and Its Cost LLM engineering and evals
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #68 Synthetic Test Cases: 27 Asked, 17 Passed Checks LLM engineering and evals
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
