learning-it by example

Back to the video

Temperature, Top-p and Seed: 20 Runs Each

I ran the same question 20 times under six settings. Two gave 20 different answers. Four gave exactly one.

Video #62 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Twenty runs, six settings
  2. 0:26 The four settings
  3. 0:55 Setup
  4. 1:07 A demo you can read
  5. 1:51 Top k and seed
  6. 2:18 The real model
  7. 2:39 Twenty runs
  8. 3:22 What this does not show
  9. 4:00 Mistakes to avoid
  10. 4:26 My take and next steps
  11. 4:55 Recap

Files

  • runs.py (25 lines)
  • sampling.py (29 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One small local model (llama3.2), one prompt, a 12-token cap and 20 runs per setting on one laptop: a demonstration, not a benchmark. The first demo uses made-up scores. Top p 0.1 was tested only at temperature 0.8, and top k 1 only at 1.5. Other models, hosted services and longer outputs were not tested. A repeatable answer is not a correct answer.

Sources

Learning IT by Example.