#318 Forward deployed engineer interviews 5.9 min
LLM Latency and Cost Budget: Which Lever Matters?
Cutting the prompt barely helped. A small model with shorter answers made it fit.
Loosen the latency limit
Copy the code folder first. In run.py change P95_LIMIT from 4.0 to 8.0. Before you run it, predict how many of the seven options will now say yes in the 'fits' column. Then run python run.py and count. Then change P95_LIMIT back to 4.0.
A hint
Compare each 95th percentile in the p95 column with 8 seconds. I counted four options that fit.
Raise the cache hit rate
In run.py change the 0.4 in the option 'big, short prompt, 40% cached' to 0.8. The label will still say 40%, which is fine. Predict whether the median or the 95th percentile will improve more, and whether the option will fit. Then run python run.py. Then change 0.8 back to 0.4.
A hint
Cache hits take 0.05 seconds and cost nothing, but the calls that miss are still slow. I got a median of 0.1s and a 95th percentile of 8.7s.
Cut the answer harder
In run.py change the last number of the option 'big, short, answers cut to 100' from 100 to 30. Predict what happens to its 95th percentile and cost, and whether it now fits. Run python run.py. Then write one sentence about what you would need to check before using such short answers. The row label still says 'answers cut to 100'. Then change 30 back to 100.
A hint
I saw 3.1s at the slow end and 0.22p, and the option fitted. Think about the video's warning on quality.
Make slow calls more common
In model.py change '1 / 30' in the call function to '1 / 10', so that one call in ten is three times slower. Predict whether the median or the 95th percentile will change more, and whether any of the seven options will fit. Run python run.py to check.
A hint
The 95th percentile looks at the slow end. In my run none of the seven options fitted, and the best 95th percentile was 5.7 seconds.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A simulation: speeds, prices, read rate and cache rate are invented; 200 calls, one seed; quality not measured.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals
- #12 Multi-Agent Patterns in Plain Python: Pipeline, Router, Loop AI agents and MCP
- #164 Batch vs Real-Time Inference: A Latency Simulation MLOps and deployment
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers LLM engineering and evals
