learning-it by example

#118 Data engineering and pipelines 6.2 min

A Durable Queue in Python: Commit Order and Crashes

With the same simulated crash at offset 2, commit-after handled order 3 twice and commit-before lost it.

1. In a simple durable queue built on an append-only log file, what is a consumer group's position?
2. After a simulated restart, a new group called audit reads from the log. What does it get?
3. The consumer crashes at offset 2 after doing the work but before committing the offset, and then restarts. What does the run show happened to order-3?
4. With the same crash at offset 2, what happened when the consumer committed the offset BEFORE doing the work?
5. Why does the commit function write a temporary file, force it to disk, and then use os.replace?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run both demos

    Copy the code folder first. Run python run.py basics and python run.py crash. In the basics output, find the offset that email committed and write down which orders email reads next. In the crash output, write down which variant handled order-3 twice and which one lost it.

    A hint

    Commit after the work can repeat a message. Commit before the work can lose one.

  2. Let email read three

    In run.py, find the line that commits email after reading with limit=2. Change the 2 to 3 and predict what python run.py basics prints for email's committed offset and for the orders it reads after the restart. On this machine email committed offset 3 and then reads order-4, order-5.

    A hint

    The committed offset is the number of messages the group has finished.

  3. Crash at a different place

    In run.py, the crash demo calls consume with crash_at=2. Change it to 4 and predict which order is handled twice in the commit after variant and which order is lost in the commit before variant. Run python run.py crash. On this machine they are order-5 in both cases, handled 6 and 4. The printed heading still says offset 2, because that text was not changed.

    A hint

    The crash happens after the message at the crash offset has been dealt with by one of the two steps. Order numbers are the offset plus 1.

  4. Add more orders

    In run.py, the ORDERS list is built with range(1, 6). Change it to range(1, 8) for seven orders. Predict what billing's offset and what audit reads in python run.py basics, and how many orders are handled in each crash variant. On this machine audit reads order-1 to order-7, and the crash run shows handled 8 for commit after and handled 6 for commit before.

    A hint

    The crash is still at offset 2, so the same order is affected. Only the number of other orders changes.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A pure-Python queue on one machine with made-up orders; the crash is an exception, not a real process kill; no broker runs, nothing was benchmarked, and Kafka and Redpanda are described only from their own docs.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos