learning-it by example

#87 Local and open models 5.5 min

Local LLM Speed: Measure Tokens per Second Honestly

One model, one laptop: 1.6, 6.6 and 9.2 tokens per second. All three were real.

1. To measure how fast a local model writes in Ollama, which speed should you quote?
2. On one CPU-only laptop that was also doing other work, the first (cold) request after unloading llama3.2 in Ollama spent 20.7 seconds loading. What were its wall-clock rate and its generating rate?
3. On one CPU-only laptop that was also doing other work, the same warm 60 token request to llama3.2 in Ollama was run five times in a row. What range of generation rates came out?
4. In a Python script that times one Ollama request, how is the generation rate worked out from the request's counters?
5. A laptop with only a processor ran the same 60 token llama3.2 request in Ollama several times. Why are those numbers not a benchmark of the processor?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Spread and median by hand

    Copy the code folder first. The video's five warm runs gave 4.3, 4.3, 3.9, 4.3 and 5.6 tokens a second. By hand, find the median, and the spread as a percentage of the slowest run using the formula in measure.py. Then check in Python: import statistics; rates = [4.3, 4.3, 3.9, 4.3, 5.6]; print(statistics.median(rates), 100 * (max(rates) - min(rates)) / min(rates)). You should get 4.3 and about 43.6, which the script prints rounded to 44%.

    A hint

    Sort the five numbers and take the middle one. The spread is (fastest minus slowest) divided by slowest, times 100.

  2. Report the median as well

    This needs Ollama with llama3.2 and takes a few minutes. In measure.py add a line after the spread line that prints the median of rates using statistics.median (add import statistics at the top). Run python measure.py. Your numbers will differ from the video. Write down your slowest, median and fastest warm rate side by side, as the video's tip suggests, and what else was running on your machine.

    A hint

    The list of five warm rates is already stored in the variable rates.

  3. Predict a longer cold test

    Predict only, then test if you like. In measure.py the cold and warm test asks for 60 tokens. If you ask for 150 tokens instead, will the cold request's wall-clock rate get closer to, or further from, its generating rate? Write your reason first. Then change generate(STORY, 60) in the first test to generate(STORY, 150), run python measure.py, and compare the two numbers on the cold line.

    A hint

    The wall-clock rate includes the load time. Think about what share of the whole request the load time is when more tokens are written.

  4. Try more prompt lengths

    In measure.py change the prompt-length loop from (1, 40) to (1, 20, 40, 80). Predict first whether the time before the first token will grow roughly in step with the prompt size, as in the video where a 33 token prompt took 1.7 seconds and a 228 token prompt took 15.1 seconds. Then run python measure.py (it needs Ollama with llama3.2) and compare the read speed in tokens a second at each length. Your numbers will differ.

    A hint

    Look at the line starting with 4 in the output. It shows the prompt tokens, the read speed and the time before the first token.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One CPU-only laptop that was also doing other work, one model (llama3.2) and five warm runs; the same warm request ranged from 3.9 to 9.2 tokens per second within one run, so these are not benchmark figures, and no graphics card was tested.

Up next in Local and open models

More in Local and open models

All Local and open models videos

Related videos