Read APIs Reliably: Pagination, Retries, Rate Limits
A local API fails 3 times on purpose, and the reliable client still gets all 23 items in 8 requests.
Video #142 ยท Python for data
Steps in the video
- 0:00 Three failures on the way, and all 23 items arrive
- 0:33 The idea
- 1:01 Setup
- 1:26 A server that fails on purpose
- 1:52 A good page
- 2:05 A free port
- 2:20 One request, with retries
- 2:52 Follow next, with a limit
- 3:10 The shortcut
- 3:26 The run
- 4:54 What this does not show
- 5:22 Mistakes to avoid
- 5:55 My take and next steps
- 6:14 Recap
Files
- client.py (44 lines)
- run.py (33 lines)
- server.py (47 lines)
Tested with
Python 3.12.15
What this does not show
A toy local API with planned failures; it does not simulate dropped connections or timeouts, handles only the seconds form of Retry-After, and measures no timing.
Sources
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/503
- https://docs.python.org/3.12/library/http.server.html
- https://docs.python.org/3.12/library/socketserver.html
- https://docs.python.org/3.12/library/urllib.error.html
- https://docs.python.org/3.12/library/urllib.request.html
- https://www.rfc-editor.org/rfc/rfc6585#section-4
- https://www.rfc-editor.org/rfc/rfc9110
Learning IT by Example.