Local LLM Speed: Measure Tokens per Second Honestly
One model, one laptop: 1.6, 6.6 and 9.2 tokens per second. All three were real.
Video #87 ยท Local and open models
Steps in the video
Files
- measure.py (30 lines)
- speed.py (38 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One CPU-only laptop that was also doing other work, one model (llama3.2) and five warm runs; the same warm request ranged from 3.9 to 9.2 tokens per second within one run, so these are not benchmark figures, and no graphics card was tested.
Sources
Learning IT by Example.