learning-it by example

#142 Python for data 6.7 min

Read APIs Reliably: Pagination, Retries, Rate Limits

A local API fails 3 times on purpose, and the reliable client still gets all 23 items in 8 requests.

1. A shortcut client asks for 10 items a page and stops when a page comes back shorter than 10. The server holds 23 made-up items and never sends more than 5 per page. How many items did the shortcut get, and why?
2. How does the reliable client know when to stop asking for pages?
3. A Python retrying client read 23 made-up items from a local mock API, with page 2 failing once with a 503, and page 4 failing with a 429 and then a 503. How many requests did it make?
4. Which HTTP errors does the retry list in the Python retry client treat as worth another try, and which does it not?
5. Page 3 fails nine times in a row, and the reliable API client allows 4 attempts. What does it do, and why is that good?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fewer planned failures

    Copy the code folder first. In run.py change FLAKY = {2: [503], 4: [429, 503]} to FLAKY = {2: [503]}, then run python run.py reliable. Predict the number of requests and the two retry counts. When I ran it, there were 6 requests, 1 retry after a 503 and 0 after a 429, with all 23 items still matching.

    A hint

    Five pages are always needed, so count the retries on top.

  2. Give up sooner

    In client.py change attempts=4 to attempts=2 in the fetch_all definition (not in get_json), then run python run.py broken. Predict how many requests the client makes before it gives up. When I ran it, it made 4 requests and retried once after a 503.

    A hint

    Pages 1 and 2 take one request each.

  3. A page limit that hides data

    In client.py change max_pages=50 to max_pages=3 in fetch_all, then run python run.py reliable. Predict how many items come back and what 'all items match' says. When I ran it, only 15 items arrived, the match line said False, and nothing raised an error. The video says real code should raise at the limit instead of returning a partial list.

    A hint

    Each page holds at most 5 items.

  4. Stop retrying a 429

    In client.py remove 429 from the RETRY_ON set, then run python run.py reliable. Predict what happens when page 4 returns a 429. When I ran it, the program stopped with an HTTP Error 429: Too Many Requests error instead of printing the item counts. Put the 429 back afterwards.

    A hint

    A code that is not in the list is raised straight away, without waiting.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A toy local API with planned failures; it does not simulate dropped connections or timeouts, handles only the seconds form of Retry-After, and measures no timing.

Up next in Python for data

More in Python for data

All Python for data videos

Related videos