Temperature, Top-p and Seed: 20 Runs Each
I ran the same question 20 times under six settings. Two gave 20 different answers. Four gave exactly one.
Video #62 ยท LLM engineering and evals
Steps in the video
Files
- runs.py (25 lines)
- sampling.py (29 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One small local model (llama3.2), one prompt, a 12-token cap and 20 runs per setting on one laptop: a demonstration, not a benchmark. The first demo uses made-up scores. Top p 0.1 was tested only at temperature 0.8, and top k 1 only at 1.5. Other models, hosted services and longer outputs were not tested. A repeatable answer is not a correct answer.
Sources
Learning IT by Example.