#142 Python for data 6.7 min
Read APIs Reliably: Pagination, Retries, Rate Limits
A local API fails 3 times on purpose, and the reliable client still gets all 23 items in 8 requests.
Fewer planned failures
Copy the code folder first. In run.py change FLAKY = {2: [503], 4: [429, 503]} to FLAKY = {2: [503]}, then run python run.py reliable. Predict the number of requests and the two retry counts. When I ran it, there were 6 requests, 1 retry after a 503 and 0 after a 429, with all 23 items still matching.
A hint
Five pages are always needed, so count the retries on top.
Give up sooner
In client.py change attempts=4 to attempts=2 in the fetch_all definition (not in get_json), then run python run.py broken. Predict how many requests the client makes before it gives up. When I ran it, it made 4 requests and retried once after a 503.
A hint
Pages 1 and 2 take one request each.
A page limit that hides data
In client.py change max_pages=50 to max_pages=3 in fetch_all, then run python run.py reliable. Predict how many items come back and what 'all items match' says. When I ran it, only 15 items arrived, the match line said False, and nothing raised an error. The video says real code should raise at the limit instead of returning a partial list.
A hint
Each page holds at most 5 items.
Stop retrying a 429
In client.py remove 429 from the RETRY_ON set, then run python run.py reliable. Predict what happens when page 4 returns a 429. When I ran it, the program stopped with an HTTP Error 429: Too Many Requests error instead of printing the item counts. Put the 429 back afterwards.
A hint
A code that is not in the list is raised straight away, without waiting.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A toy local API with planned failures; it does not simulate dropped connections or timeouts, handles only the seconds form of Retry-After, and measures no timing.
Up next in Python for data
More in Python for data
- #08 Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked
- #09 Clean Messy Data with Pandas 3: End to End, 3 Mistakes
- #137 Polars for pandas Users: One Report, Translated
- #138 Seven pandas Changes Tested on 1M Rows: Speed and Memory
- #139 Validate Data with Pydantic: Catch Bad Rows
- #140 Python Dates and Time Zones: 5 Bugs Reproduced