learning-it by example

#15 Data engineering and pipelines 5.7 min

Kafka Streaming Basics in Python with a Redpanda Broker

Run a Kafka-compatible broker in Docker, stream ten events from Python, read them back, and see why a consumer can show nothing.

1. In a streaming topic with several partitions, what does an event's key do?
2. The producer sent 10 order events for Ava, Ben and Cleo. What totals did the consumer print?
3. The same consumer was run a second time against the same topic and read 0 events. Why?
4. What does setting auto.offset.reset to earliest do for a consumer?
5. Why does the producer script end with a call to flush?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Halve the amounts

    Copy the code folder first. Start a fresh broker (docker compose up -d --wait, then create the orders topic with three partitions as in the video). In producer.py change 10 * (n + 1) to 5 * (n + 1). Predict the three totals before you run the producer and then the consumer. Stop with docker compose down -v afterwards. This needs Docker and confluent-kafka.

    A hint

    Every total is half of the video's total, because every amount is half.

  2. Read everything again

    On a fresh broker (start it and create the orders topic as in exercise 1), run the producer and the consumer once. After the consumer has read the ten events and printed its totals, change group.id in consumer.py from totals to totals2 and run it again. Predict how many events it reads this time and explain why, then run it.

    A hint

    The video says to read again, use a new group name or reset the group's offsets.

  3. Add a fourth customer

    On a fresh broker, add 'Dev' to the CUSTOMERS list in producer.py and change n % 3 to n % len(CUSTOMERS). Predict the four totals the consumer will print, and whether each customer still stays in one partition. Then run the producer and the consumer.

    A hint

    The amount for event n is 10 * (n + 1). Work out which events each customer gets.

  4. Read only five events

    On a fresh broker run the producer. In consumer.py change while seen < 10 to while seen < 5, and run the consumer twice. Predict how many events each run reads, using the fact that closing the consumer saves its place.

    A hint

    The group remembers how far it has read.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

The broker is Redpanda, which speaks the Kafka protocol but is not Apache Kafka, so edge cases can differ. One broker, one laptop, ten made-up events: no replication, no failure test, no load. You need Docker to follow along.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos