learning-it by example

#62 LLM engineering and evals 5.4 min

Temperature, Top-p and Seed: 20 Runs Each

I ran the same question 20 times under six settings. Two gave 20 different answers. Four gave exactly one.

1. What does the temperature setting do to the next word odds when a model picks words?
2. In the made up demo of six words, what share did the top word 'great' get at temperature 0.5, 1 and 2?
3. In the video's real-model test (llama3.2 in Ollama, one prompt about Mondays, a 12-token cap, 20 runs per setting), which settings gave 20 different answers and which gave exactly 1?
4. What surprise did the video find with llama3.2 in Ollama when it set a low top p of 0.1 at temperature 0.8, or top k 1 at temperature 1.5?
5. In the video's llama3.2 test, did a seed give the same text as temperature zero, and does a repeatable answer mean a correct answer?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Try new temperatures in the demo

    Copy the code folder first. In sampling.py change the loop (0.5, 1.0, 2.0) to (0.1, 1.0, 5.0). Predict what the top word 'great' gets at 0.1 and at 5.0, then run python sampling.py. You should see great at 1.00 for temperature 0.1 and at 0.23 for temperature 5.0, with the six words much closer together at 5.0, from 0.23 down to 0.11. No model is needed.

    A hint

    Low temperature makes the likeliest word even more likely, and high temperature spreads the odds out.

  2. Change the top p values

    In sampling.py change the last loop (0.9, 0.5) to (0.7, 0.99). Predict which words each value keeps, then run python sampling.py. Top p 0.7 should keep great and good, and top p 0.99 should keep all six words. The first keeps only two words because the top two add up to 0.75, which is above 0.7; the second needs every word because the first five only add up to about 0.98.

    A hint

    The nucleus function adds up the chances from the most likely word down and stops once the total reaches p.

  3. Make a different word the favourite

    In sampling.py change the score for blue from -0.5 to 3.0, so it ties with great. Predict how the odds at temperature 1.0 change, then run python sampling.py. You should see great and blue each at 0.35 at temperature 1.0, and top p 0.5 keeping both of them.

    A hint

    Two words with equal scores get equal odds at every temperature.

  4. Run the real model with your own setting

    Add a new line to SETTINGS in runs.py, such as 'warm': {'temperature': 1.0}. Predict how many different answers you will get in 20 runs before you start. If Ollama and llama3.2 are installed, run python runs.py warm 20 and compare with the video, where the default gave 20 different answers in 20 runs.

    A hint

    Change one setting at a time. If you change two, you cannot tell which one made the difference.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One small local model (llama3.2), one prompt, a 12-token cap and 20 runs per setting on one laptop: a demonstration, not a benchmark. The first demo uses made-up scores. Top p 0.1 was tested only at temperature 0.8, and top k 1 only at 1.5. Other models, hosted services and longer outputs were not tested. A repeatable answer is not a correct answer.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos