#162 MLOps and deployment 6.6 min
Cost Control for AI Apps: A Token Meter With Caps
In a made-up day, a 40-cent user cap and a 60-cent daily cap each refused a call before it ran.
Raise the daily cap
Copy the code folder first. In meter.py change DAY_CAP from 600_000 to 1_000_000. Predict which refusals disappear, then run python run.py log. On my machine cara's second call was allowed with a real cost of $0.108, and ben's third call was still refused by the user cap.
A hint
The daily cap is checked after the user cap.
Raise the user cap instead
Put DAY_CAP back to 600_000 and change USER_CAP from 400_000 to 600_000. Predict who is refused now, then run python run.py log. On my machine ben's third call was allowed, and ana's second call and cara's second call were both refused by the daily cap.
A hint
Add up the day's real costs as the calls are allowed.
Check a worst case yourself
Put USER_CAP back to 400_000. In run.py change cara's second request from (1, "cara", 3000, 1000) to (1, "cara", 3000, 600). First work out its worst case in micro-dollars (input tokens times 20 plus the output limit times 80), then run python run.py log. On my machine the worst case was $0.108 and the call was allowed.
A hint
Dollars are micro-dollars divided by 1,000,000.
Make the output cheaper
In meter.py change the output price from 80 to 40 micro-dollars per token. Predict whether anything is refused and how much day 1 spends, then run python run.py log and python run.py totals. On my machine nothing was refused and day 1 spent $0.550.
A hint
Both the worst case and the real cost use the same price table.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
No model or API is called: the model is a stand-in, tokens are counted as words, prices and users are made up, and the totals live in memory only.
Up next in MLOps and deployment
More in MLOps and deployment
Related videos
- #244 Cost Control for AI Agents: A Budget for Each Agent Enterprise architecture and AI agents
- #61 Count Tokens Before You Send: Context Limits LLM engineering and evals
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search