learning-it by example

Back to the video

Cut LLM Cost and Latency: Caching and Shorter Answers

Same six requests, over two minutes on a laptop. Add a cache and shorter answers, and watch the output tokens and the wait drop, with the real numbers on screen.

Video #17 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Cost and wait depend on one number
  2. 0:22 Why output tokens matter
  3. 0:56 Setup
  4. 1:23 The baseline
  5. 2:05 Fix one: a cache
  6. 2:57 Fix two: shorter answers
  7. 3:42 What this does not show
  8. 4:11 Mistakes to avoid
  9. 4:42 My take and next steps
  10. 5:05 Recap

Files

  • baseline.py (18 lines)
  • cached.py (35 lines)
  • concise.py (18 lines)
  • traffic.py (7 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Six made-up requests (three questions, each asked twice) on one small local model, one run on one laptop. Tokens stand in for cost because a local model is free, so no bill was measured. The cache only helps exact repeats, shorter answers can lose detail, and the two fixes were not tested together. The two advice articles are vendor blog posts.

Sources

Learning IT by Example.