learning-it by example

Back to the video

A Durable Queue in Python: Commit Order and Crashes

With the same simulated crash at offset 2, commit-after handled order 3 twice and commit-before lost it.

Video #118 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 Same simulated crash, two commit orders: an order handled twice, or lost
  2. 0:34 The idea
  3. 1:01 Setup
  4. 1:15 Append
  5. 1:31 Read
  6. 1:42 Commit
  7. 2:04 Groups
  8. 2:46 The crash
  9. 3:46 What the brokers add
  10. 4:23 What this does not show
  11. 4:45 Mistakes to avoid
  12. 5:09 My take and next steps
  13. 5:43 Recap

Files

  • consumer.py (19 lines)
  • logqueue.py (38 lines)
  • run.py (42 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A pure-Python queue on one machine with made-up orders; the crash is an exception, not a real process kill; no broker runs, nothing was benchmarked, and Kafka and Redpanda are described only from their own docs.

Sources

Learning IT by Example.