LLM Latency and Cost Budget: Which Lever Matters?
Cutting the prompt barely helped. A small model with shorter answers made it fit.
Video #318 ยท Forward deployed engineer interviews
Steps in the video
Files
- budget.py (17 lines)
- model.py (31 lines)
- run.py (21 lines)
Tested with
Python 3.12.15
What this does not show
A simulation: speeds, prices, read rate and cache rate are invented; 200 calls, one seed; quality not measured.
Sources
Learning IT by Example.