learning-it by example

#63 LLM engineering and evals 5.6 min

Few-Shot Examples: How Many Is Enough? A Measured Test

Zero examples scored 12 of 20. Eight scored 18. Flipped labels dropped it to 13.

1. What is a few-shot prompt?
2. In the video's test of llama3.2 on twenty made-up support messages (urgent or normal by a made-up house rule), how many were right with 0, 2, 4 and 8 examples in the prompt?
3. In the video's test of llama3.2 on twenty made-up support messages, what happened when the same four examples were given as written, reversed and rotated?
4. In the video's test of llama3.2 on twenty made-up support messages, what happened to the score when every label in the eight examples was flipped, and why does that not contradict the paper the video cited?
5. In the video's test of llama3.2 on twenty made-up support messages, which two test messages did the 8-example prompt still get wrong, and what label did the model give them?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the house rule from the examples

    Copy the code folder first. Open examples.py and read the eight EXAMPLES. In your own words, write the house rule that the labels show, and say which of the eight examples show that anger is not urgent. The prompt never states the rule, so only the examples teach it.

    A hint

    The video said money taken wrongly, account security and safety are urgent, and anger is not.

  2. Print the prompts and read them

    Run python -c with this idea, or add a small script: import build from classify and EXAMPLES from examples, then print build('Hello there', EXAMPLES[:2]) and print(build('Hello there', [])). The first prompt should hold the instruction, two examples that each end in a label, then 'Message: Hello there' and 'Label:'. The second should hold only the instruction and the new message. Check that this is what you see.

    A hint

    You need the ollama package installed for the import to work, but this does not call the model.

  3. Predict a score with your own examples

    Write down which two of the eight examples you would choose if you could only use two, and predict whether the score will beat the 16 of 20 that the first two examples got in the video. If Ollama and llama3.2 are installed, write a small script that imports accuracy from classify and TESTS and EXAMPLES from examples, and call accuracy([EXAMPLES[i], EXAMPLES[j]], TESTS) for your two examples.

    A hint

    Keep an urgent example and a normal example, so the pair stays balanced.

  4. Add a trap message of your own

    Add a twenty-first tuple to TESTS in examples.py, a message that sounds urgent but is normal by the house rule, such as a very angry message about a missing leaflet. Predict whether the 8-example prompt will get it right. If you can run Ollama, run python order.py and look at whether your message appears in the misses list.

    A hint

    The video said the traps are what you need to check, not only the easy cases.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Twenty made-up test messages and eight made-up examples, all written by the author in one style, one made-up rule, one small local model (llama3.2), one run at temperature 0. A gap of one or two answers is within noise. Order was tested with three orders of one set of four examples. The label test flipped every label, which is not the random replacement tested in the cited paper. Not a benchmark.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos