#118 Data engineering and pipelines 6.2 min
A Durable Queue in Python: Commit Order and Crashes
With the same simulated crash at offset 2, commit-after handled order 3 twice and commit-before lost it.
Run both demos
Copy the code folder first. Run python run.py basics and python run.py crash. In the basics output, find the offset that email committed and write down which orders email reads next. In the crash output, write down which variant handled order-3 twice and which one lost it.
A hint
Commit after the work can repeat a message. Commit before the work can lose one.
Let email read three
In run.py, find the line that commits email after reading with limit=2. Change the 2 to 3 and predict what python run.py basics prints for email's committed offset and for the orders it reads after the restart. On this machine email committed offset 3 and then reads order-4, order-5.
A hint
The committed offset is the number of messages the group has finished.
Crash at a different place
In run.py, the crash demo calls consume with crash_at=2. Change it to 4 and predict which order is handled twice in the commit after variant and which order is lost in the commit before variant. Run python run.py crash. On this machine they are order-5 in both cases, handled 6 and 4. The printed heading still says offset 2, because that text was not changed.
A hint
The crash happens after the message at the crash offset has been dealt with by one of the two steps. Order numbers are the offset plus 1.
Add more orders
In run.py, the ORDERS list is built with range(1, 6). Change it to range(1, 8) for seven orders. Predict what billing's offset and what audit reads in python run.py basics, and how many orders are handled in each crash variant. On this machine audit reads order-1 to order-7, and the crash run shows handled 8 for commit after and handled 6 for commit before.
A hint
The crash is still at offset 2, so the same order is affected. Only the number of other orders changes.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A pure-Python queue on one machine with made-up orders; the crash is an exception, not a real process kill; no broker runs, nothing was benchmarked, and Kafka and Redpanda are described only from their own docs.
Up next in Data engineering and pipelines
More in Data engineering and pipelines
Related videos
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results RAG and vector search
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON LLM engineering and evals
