learning-it by example

Back to the video

Local LLM Speed: Measure Tokens per Second Honestly

One model, one laptop: 1.6, 6.6 and 9.2 tokens per second. All three were real.

Video #87 ยท Local and open models

Steps in the video

  1. 0:00 Three speeds
  2. 0:30 The idea
  3. 1:03 Setup
  4. 1:24 One timed request
  5. 1:43 Two rates
  6. 2:03 Cold, warm and repeated
  7. 2:20 Length of output and prompt
  8. 3:29 What this does not show
  9. 4:02 Mistakes to avoid
  10. 4:23 My take and next steps
  11. 4:52 Recap

Files

  • measure.py (30 lines)
  • speed.py (38 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One CPU-only laptop that was also doing other work, one model (llama3.2) and five warm runs; the same warm request ranged from 3.9 to 9.2 tokens per second within one run, so these are not benchmark figures, and no graphics card was tested.

Sources

Learning IT by Example.