Cut LLM Cost and Latency: Caching and Shorter Answers
Same six requests, over two minutes on a laptop. Add a cache and shorter answers, and watch the output tokens and the wait drop, with the real numbers on screen.
Video #17 ยท LLM engineering and evals
Steps in the video
Files
- baseline.py (18 lines)
- cached.py (35 lines)
- concise.py (18 lines)
- traffic.py (7 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Six made-up requests (three questions, each asked twice) on one small local model, one run on one laptop. Tokens stand in for cost because a local model is free, so no bill was measured. The cache only helps exact repeats, shorter answers can lose detail, and the two fixes were not tested together. The two advice articles are vendor blog posts.
Sources
Learning IT by Example.