#91 AI security and governance 5.3 min
Rate Limiting an LLM API: Token Bucket in FastAPI
In my test, one simulated user sent 400 big requests in 20 seconds, and my API answered all of them.
Work out why Mallory gets 11
Copy the code folder first. In compare.py the limiter is TokenBucket(400, 2000, ...): it refills 400 tokens a second and holds at most 2000. By hand, work out the most tokens a user could spend in the 20 seconds if the bucket starts full: the starting 2000 plus 20 seconds of refill. Divide by Mallory's 900 tokens a request and round down. Compare your answer with the 11 of 400 in the video.
A hint
The total is the starting 2000 plus 400 times 20, because the bucket can never hold or give out more than that.
Double the refill rate
In compare.py change the 400 in TokenBucket(400, 2000, ...) to 800. Predict first how many of Mallory's 400 requests will now get through with a bucket for each user, and whether Ana and Ben are affected. Then run python compare.py. For the last line you should see Ana 20 of 20, Ben 20 of 20 and Mallory 19 of 400. What does the refill rate control, and why did Ana and Ben not change?
A hint
Use the same sum as the first task with the new refill rate, but Mallory's last request arrives at 19.95 seconds, not 20: 2000 + 800 x 19.95 = 17,960 tokens, which is 19 requests of 900 (with 400 the same sum gives 11). Ana and Ben each send only 150 tokens a second, far below the refill rate.
Make Mallory's requests small
First put the 800 in TokenBucket back to 400. Then in compare.py change Mallory's 900 in the TRAFFIC list to 150, so that her requests cost the same as Ana's. Predict first how many of her 400 requests get through with a bucket for each user. Then run python compare.py and look at the last line. You should see Ana and Ben at 20 of 20 and Mallory at 66 of 400. Why does she get far more through than with 900 token requests, even though she still sends 20 a second?
A hint
The limit counts tokens, not requests. Divide the most tokens she can spend by 150.
Read a refusal as a client
Predict only. In service.py find the line that raises the refusal. Write down the status code and the header value that a client would receive when its request is refused. Then write two or three sentences for what a well behaved client should do on getting that reply, and why retrying at once would make things worse. The video lists retrying at once after a 429 as a mistake.
A hint
The Retry-After header tells the client how many seconds to wait.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A fake clock, three made-up users and numbers I chose; no model and no server run; the user and the token count come from headers the client sets; the buckets live in the memory of one process.
Up next in AI security and governance
More in AI security and governance
Related videos
- #230 Agent Gateway: One Front Door for Models, Tools, Agents Enterprise architecture and AI agents
- #89 Jailbreak-Test Your Own App: Eight Wrappers, Counted AI security and governance
- #90 Stop Leaking API Keys: Build a Secret Scanner in Python AI security and governance
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
