learning-it by example

Back to the video

LLM Latency and Cost Budget: Which Lever Matters?

Cutting the prompt barely helped. A small model with shorter answers made it fit.

Video #318 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 Shortening the prompt barely helped
  2. 0:38 The idea
  3. 1:24 Setup
  4. 1:50 One call
  5. 2:16 Two hundred calls
  6. 2:37 The budget, as a test
  7. 3:46 What this does not show
  8. 4:24 Mistakes to avoid
  9. 4:49 My take and next steps
  10. 5:20 Recap

Files

  • budget.py (17 lines)
  • model.py (31 lines)
  • run.py (21 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A simulation: speeds, prices, read rate and cache rate are invented; 200 calls, one seed; quality not measured.

Sources

Learning IT by Example.