learning-it by example

#91 AI security and governance 5.3 min

Rate Limiting an LLM API: Token Bucket in FastAPI

In my test, one simulated user sent 400 big requests in 20 seconds, and my API answered all of them.

  1. Work out why Mallory gets 11

    Copy the code folder first. In compare.py the limiter is TokenBucket(400, 2000, ...): it refills 400 tokens a second and holds at most 2000. By hand, work out the most tokens a user could spend in the 20 seconds if the bucket starts full: the starting 2000 plus 20 seconds of refill. Divide by Mallory's 900 tokens a request and round down. Compare your answer with the 11 of 400 in the video.

    A hint

    The total is the starting 2000 plus 400 times 20, because the bucket can never hold or give out more than that.

  2. Double the refill rate

    In compare.py change the 400 in TokenBucket(400, 2000, ...) to 800. Predict first how many of Mallory's 400 requests will now get through with a bucket for each user, and whether Ana and Ben are affected. Then run python compare.py. For the last line you should see Ana 20 of 20, Ben 20 of 20 and Mallory 19 of 400. What does the refill rate control, and why did Ana and Ben not change?

    A hint

    Use the same sum as the first task with the new refill rate, but Mallory's last request arrives at 19.95 seconds, not 20: 2000 + 800 x 19.95 = 17,960 tokens, which is 19 requests of 900 (with 400 the same sum gives 11). Ana and Ben each send only 150 tokens a second, far below the refill rate.

  3. Make Mallory's requests small

    First put the 800 in TokenBucket back to 400. Then in compare.py change Mallory's 900 in the TRAFFIC list to 150, so that her requests cost the same as Ana's. Predict first how many of her 400 requests get through with a bucket for each user. Then run python compare.py and look at the last line. You should see Ana and Ben at 20 of 20 and Mallory at 66 of 400. Why does she get far more through than with 900 token requests, even though she still sends 20 a second?

    A hint

    The limit counts tokens, not requests. Divide the most tokens she can spend by 150.

  4. Read a refusal as a client

    Predict only. In service.py find the line that raises the refusal. Write down the status code and the header value that a client would receive when its request is refused. Then write two or three sentences for what a well behaved client should do on getting that reply, and why retrying at once would make things worse. The video lists retrying at once after a 429 as a mistake.

    A hint

    The Retry-After header tells the client how many seconds to wait.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A fake clock, three made-up users and numbers I chose; no model and no server run; the user and the token count come from headers the client sets; the buckets live in the memory of one process.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos