#15 Data engineering and pipelines 5.7 min
Kafka Streaming Basics in Python with a Redpanda Broker
Run a Kafka-compatible broker in Docker, stream ten events from Python, read them back, and see why a consumer can show nothing.
Halve the amounts
Copy the code folder first. Start a fresh broker (docker compose up -d --wait, then create the orders topic with three partitions as in the video). In producer.py change 10 * (n + 1) to 5 * (n + 1). Predict the three totals before you run the producer and then the consumer. Stop with docker compose down -v afterwards. This needs Docker and confluent-kafka.
A hint
Every total is half of the video's total, because every amount is half.
Read everything again
On a fresh broker (start it and create the orders topic as in exercise 1), run the producer and the consumer once. After the consumer has read the ten events and printed its totals, change group.id in consumer.py from totals to totals2 and run it again. Predict how many events it reads this time and explain why, then run it.
A hint
The video says to read again, use a new group name or reset the group's offsets.
Add a fourth customer
On a fresh broker, add 'Dev' to the CUSTOMERS list in producer.py and change n % 3 to n % len(CUSTOMERS). Predict the four totals the consumer will print, and whether each customer still stays in one partition. Then run the producer and the consumer.
A hint
The amount for event n is 10 * (n + 1). Work out which events each customer gets.
Read only five events
On a fresh broker run the producer. In consumer.py change while seen < 10 to while seen < 5, and run the consumer twice. Predict how many events each run reads, using the fact that closing the consumer saves its place.
A hint
The group remembers how far it has read.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The broker is Redpanda, which speaks the Kafka protocol but is not Apache Kafka, so edge cases can differ. One broker, one laptop, ten made-up events: no replication, no failure test, no load. You need Docker to follow along.
Up next in Data engineering and pipelines
More in Data engineering and pipelines
Related videos
- #07 SQL Window Functions and CTEs in DuckDB: Two Mistakes SQL and analytics
- #08 Pandas vs Polars vs DuckDB on 10M Rows, Timed and Checked Python for data
- #145 Type Hints and mypy: Catch Data Bugs Before You Run Python for data
- #150 Read Big CSVs in Small Memory: Stream and Chunk in Python Python for data
