Rate Limiting an LLM API: Token Bucket in FastAPI
In my test, one simulated user sent 400 big requests in 20 seconds, and my API answered all of them.
Video #91 ยท AI security and governance
Steps in the video
Files
- compare.py (34 lines)
- limiter.py (19 lines)
- service.py (14 lines)
Tested with
Python 3.12.15 | fastapi==0.161.1, starlette==1.9.0, httpx==0.28.1
What this does not show
A fake clock, three made-up users and numbers I chose; no model and no server run; the user and the token count come from headers the client sets; the buckets live in the memory of one process.
Sources
Learning IT by Example.