learning-it by example

Back to the video

Kafka Streaming Basics in Python with a Redpanda Broker

Run a Kafka-compatible broker in Docker, stream ten events from Python, read them back, and see why a consumer can show nothing.

Video #15 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 Streaming data on a laptop
  2. 0:22 Five words to know
  3. 0:53 Setup
  4. 1:14 The broker
  5. 1:57 The producer
  6. 2:21 The consumer
  7. 3:04 Why my consumer sees nothing
  8. 3:45 What this does not show
  9. 4:18 Mistakes to avoid
  10. 4:53 My take and next steps
  11. 5:17 Recap

Files

  • compose.yaml (20 lines)
  • consumer.py (23 lines)
  • producer.py (13 lines)

Download the code (zip)

Tested with

Python 3.12.15 | confluent-kafka==2.16.1

What this does not show

The broker is Redpanda, which speaks the Kafka protocol but is not Apache Kafka, so edge cases can differ. One broker, one laptop, ten made-up events: no replication, no failure test, no load. You need Docker to follow along.

Sources

Learning IT by Example.