learning-it by example

#318 Forward deployed engineer interviews 5.9 min

LLM Latency and Cost Budget: Which Lever Matters?

Cutting the prompt barely helped. A small model with shorter answers made it fit.

1. Why does the video write a latency budget with the 95th percentile rather than the average?
2. In the seven-option simulation, with a budget of 95th percentile under 4 seconds and cost under 0.4p a request, which option fitted?
3. What did the cache with a 40% hit rate do for the big model with a short prompt in the LLM latency and cost budget simulation?
4. What does the video warn about when answers are cut to 100 tokens to meet a budget?
5. In the simulated call, what makes one call in 30 take three times longer?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Loosen the latency limit

    Copy the code folder first. In run.py change P95_LIMIT from 4.0 to 8.0. Before you run it, predict how many of the seven options will now say yes in the 'fits' column. Then run python run.py and count. Then change P95_LIMIT back to 4.0.

    A hint

    Compare each 95th percentile in the p95 column with 8 seconds. I counted four options that fit.

  2. Raise the cache hit rate

    In run.py change the 0.4 in the option 'big, short prompt, 40% cached' to 0.8. The label will still say 40%, which is fine. Predict whether the median or the 95th percentile will improve more, and whether the option will fit. Then run python run.py. Then change 0.8 back to 0.4.

    A hint

    Cache hits take 0.05 seconds and cost nothing, but the calls that miss are still slow. I got a median of 0.1s and a 95th percentile of 8.7s.

  3. Cut the answer harder

    In run.py change the last number of the option 'big, short, answers cut to 100' from 100 to 30. Predict what happens to its 95th percentile and cost, and whether it now fits. Run python run.py. Then write one sentence about what you would need to check before using such short answers. The row label still says 'answers cut to 100'. Then change 30 back to 100.

    A hint

    I saw 3.1s at the slow end and 0.22p, and the option fitted. Think about the video's warning on quality.

  4. Make slow calls more common

    In model.py change '1 / 30' in the call function to '1 / 10', so that one call in ten is three times slower. Predict whether the median or the 95th percentile will change more, and whether any of the seven options will fit. Run python run.py to check.

    A hint

    The 95th percentile looks at the slow end. In my run none of the seven options fitted, and the best 95th percentile was 5.7 seconds.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A simulation: speeds, prices, read rate and cache rate are invented; 200 calls, one seed; quality not measured.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos