learning-it by example

Back to the video

Rate Limiting an LLM API: Token Bucket in FastAPI

In my test, one simulated user sent 400 big requests in 20 seconds, and my API answered all of them.

Video #91 ยท AI security and governance

Steps in the video

  1. 0:00 One user used almost everything, until the limit
  2. 0:31 The idea
  3. 1:05 Setup
  4. 1:31 The bucket
  5. 1:59 The service
  6. 2:25 Two polite users and a flood
  7. 3:19 What this does not show
  8. 3:55 Mistakes to avoid
  9. 4:25 My take and next steps
  10. 4:52 Recap

Files

  • compare.py (34 lines)
  • limiter.py (19 lines)
  • service.py (14 lines)

Download the code (zip)

Tested with

Python 3.12.15 | fastapi==0.161.1, starlette==1.9.0, httpx==0.28.1

What this does not show

A fake clock, three made-up users and numbers I chose; no model and no server run; the user and the token count come from headers the client sets; the buckets live in the memory of one process.

Sources

Learning IT by Example.