#87 Local and open models 5.5 min
Local LLM Speed: Measure Tokens per Second Honestly
One model, one laptop: 1.6, 6.6 and 9.2 tokens per second. All three were real.
Spread and median by hand
Copy the code folder first. The video's five warm runs gave 4.3, 4.3, 3.9, 4.3 and 5.6 tokens a second. By hand, find the median, and the spread as a percentage of the slowest run using the formula in measure.py. Then check in Python: import statistics; rates = [4.3, 4.3, 3.9, 4.3, 5.6]; print(statistics.median(rates), 100 * (max(rates) - min(rates)) / min(rates)). You should get 4.3 and about 43.6, which the script prints rounded to 44%.
A hint
Sort the five numbers and take the middle one. The spread is (fastest minus slowest) divided by slowest, times 100.
Report the median as well
This needs Ollama with llama3.2 and takes a few minutes. In measure.py add a line after the spread line that prints the median of rates using statistics.median (add import statistics at the top). Run python measure.py. Your numbers will differ from the video. Write down your slowest, median and fastest warm rate side by side, as the video's tip suggests, and what else was running on your machine.
A hint
The list of five warm rates is already stored in the variable rates.
Predict a longer cold test
Predict only, then test if you like. In measure.py the cold and warm test asks for 60 tokens. If you ask for 150 tokens instead, will the cold request's wall-clock rate get closer to, or further from, its generating rate? Write your reason first. Then change generate(STORY, 60) in the first test to generate(STORY, 150), run python measure.py, and compare the two numbers on the cold line.
A hint
The wall-clock rate includes the load time. Think about what share of the whole request the load time is when more tokens are written.
Try more prompt lengths
In measure.py change the prompt-length loop from (1, 40) to (1, 20, 40, 80). Predict first whether the time before the first token will grow roughly in step with the prompt size, as in the video where a 33 token prompt took 1.7 seconds and a 228 token prompt took 15.1 seconds. Then run python measure.py (it needs Ollama with llama3.2) and compare the read speed in tokens a second at each length. Your numbers will differ.
A hint
Look at the line starting with 4 in the output. It shows the prompt tokens, the read speed and the time before the first token.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One CPU-only laptop that was also doing other work, one model (llama3.2) and five warm runs; the same warm request ranged from 3.9 to 9.2 tokens per second within one run, so these are not benchmark figures, and no graphics card was tested.
Up next in Local and open models
More in Local and open models
Related videos
- #83 Context Length vs Memory: Why Local LLMs Slow Down Local and open models
- #26 Plan, Act, Reflect: Does It Help a Small Model? AI agents and MCP
- #47 6 RAG Failure Modes Reproduced: Which Ones Actually Break RAG and vector search
- #51 Multi-User RAG: Filter Before You Search (Leak Test) RAG and vector search
