A Durable Queue in Python: Commit Order and Crashes
With the same simulated crash at offset 2, commit-after handled order 3 twice and commit-before lost it.
Video #118 ยท Data engineering and pipelines
Steps in the video
Files
- consumer.py (19 lines)
- logqueue.py (38 lines)
- run.py (42 lines)
Tested with
Python 3.12.15
What this does not show
A pure-Python queue on one machine with made-up orders; the crash is an exception, not a real process kill; no broker runs, nothing was benchmarked, and Kafka and Redpanda are described only from their own docs.
Sources
Learning IT by Example.