#62 LLM engineering and evals 5.4 min
Temperature, Top-p and Seed: 20 Runs Each
I ran the same question 20 times under six settings. Two gave 20 different answers. Four gave exactly one.
Try new temperatures in the demo
Copy the code folder first. In sampling.py change the loop (0.5, 1.0, 2.0) to (0.1, 1.0, 5.0). Predict what the top word 'great' gets at 0.1 and at 5.0, then run python sampling.py. You should see great at 1.00 for temperature 0.1 and at 0.23 for temperature 5.0, with the six words much closer together at 5.0, from 0.23 down to 0.11. No model is needed.
A hint
Low temperature makes the likeliest word even more likely, and high temperature spreads the odds out.
Change the top p values
In sampling.py change the last loop (0.9, 0.5) to (0.7, 0.99). Predict which words each value keeps, then run python sampling.py. Top p 0.7 should keep great and good, and top p 0.99 should keep all six words. The first keeps only two words because the top two add up to 0.75, which is above 0.7; the second needs every word because the first five only add up to about 0.98.
A hint
The nucleus function adds up the chances from the most likely word down and stops once the total reaches p.
Make a different word the favourite
In sampling.py change the score for blue from -0.5 to 3.0, so it ties with great. Predict how the odds at temperature 1.0 change, then run python sampling.py. You should see great and blue each at 0.35 at temperature 1.0, and top p 0.5 keeping both of them.
A hint
Two words with equal scores get equal odds at every temperature.
Run the real model with your own setting
Add a new line to SETTINGS in runs.py, such as 'warm': {'temperature': 1.0}. Predict how many different answers you will get in 20 runs before you start. If Ollama and llama3.2 are installed, run python runs.py warm 20 and compare with the video, where the default gave 20 different answers in 20 runs.
A hint
Change one setting at a time. If you change two, you cannot tell which one made the difference.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One small local model (llama3.2), one prompt, a 12-token cap and 20 runs per setting on one laptop: a demonstration, not a benchmark. The first demo uses made-up scores. Top p 0.1 was tested only at temperature 0.8, and top k 1 only at 1.5. Other models, hosted services and longer outputs were not tested. A repeatable answer is not a correct answer.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #12 Multi-Agent Patterns in Plain Python: Pipeline, Router, Loop AI agents and MCP
- #25 Tool Errors: Retries, Fallbacks and Made-Up Answers AI agents and MCP
- #318 LLM Latency and Cost Budget: Which Lever Matters? Forward deployed engineer interviews
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals
