learning-it by example

#162 MLOps and deployment 6.6 min

Cost Control for AI Apps: A Token Meter With Caps

In a made-up day, a 40-cent user cap and a 60-cent daily cap each refused a call before it ran.

1. How does the video's cost meter decide whether to allow an AI call?
2. In a made-up day of the Python cost meter (40 cent user cap, 60 cent daily cap), ben's third big request was refused by the 40 cent user cap. Why?
3. Cara's second call was refused by the daily cap, although its real cost would have fitted. Why?
4. Why does the video keep money as whole micro-dollars instead of floats?
5. What does the video say about where the meter's totals live?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Raise the daily cap

    Copy the code folder first. In meter.py change DAY_CAP from 600_000 to 1_000_000. Predict which refusals disappear, then run python run.py log. On my machine cara's second call was allowed with a real cost of $0.108, and ben's third call was still refused by the user cap.

    A hint

    The daily cap is checked after the user cap.

  2. Raise the user cap instead

    Put DAY_CAP back to 600_000 and change USER_CAP from 400_000 to 600_000. Predict who is refused now, then run python run.py log. On my machine ben's third call was allowed, and ana's second call and cara's second call were both refused by the daily cap.

    A hint

    Add up the day's real costs as the calls are allowed.

  3. Check a worst case yourself

    Put USER_CAP back to 400_000. In run.py change cara's second request from (1, "cara", 3000, 1000) to (1, "cara", 3000, 600). First work out its worst case in micro-dollars (input tokens times 20 plus the output limit times 80), then run python run.py log. On my machine the worst case was $0.108 and the call was allowed.

    A hint

    Dollars are micro-dollars divided by 1,000,000.

  4. Make the output cheaper

    In meter.py change the output price from 80 to 40 micro-dollars per token. Predict whether anything is refused and how much day 1 spends, then run python run.py log and python run.py totals. On my machine nothing was refused and day 1 spent $0.550.

    A hint

    Both the worst case and the real cost use the same price table.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No model or API is called: the model is a stand-in, tokens are counted as words, prices and users are made up, and the totals live in memory only.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos