learning-it by example

Back to the video

Context Length vs Memory: Why Local LLMs Slow Down

Reading a 2,542 token prompt took my laptop 155 seconds.

Video #83 ยท Local and open models

Steps in the video

  1. 0:00 Why it slows down
  2. 0:31 The idea
  3. 1:06 Setup
  4. 1:25 The formula
  5. 2:13 What Ollama reports
  6. 3:08 Timing the prompt
  7. 4:04 What this does not show
  8. 4:34 Mistakes to avoid
  9. 4:56 My take and next steps
  10. 5:16 Recap

Files

  • kv.py (14 lines)
  • measure.py (14 lines)
  • memory.py (10 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

One CPU-only laptop that was also doing other work, one model (llama3.2) and one run per prompt length; the memory figures are what Ollama reports, the gap to my cache formula is unexplained, and no graphics card, smaller cache type or flash attention was tested.

Sources

Learning IT by Example.