Context Length vs Memory: Why Local LLMs Slow Down
Reading a 2,542 token prompt took my laptop 155 seconds.
Video #83 ยท Local and open models
Steps in the video
Files
- kv.py (14 lines)
- measure.py (14 lines)
- memory.py (10 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
One CPU-only laptop that was also doing other work, one model (llama3.2) and one run per prompt length; the memory figures are what Ollama reports, the gap to my cache formula is unexplained, and no graphics card, smaller cache type or flash attention was tested.
Sources
Learning IT by Example.