Kafka Streaming Basics in Python with a Redpanda Broker
Run a Kafka-compatible broker in Docker, stream ten events from Python, read them back, and see why a consumer can show nothing.
Video #15 ยท Data engineering and pipelines
Steps in the video
Files
- compose.yaml (20 lines)
- consumer.py (23 lines)
- producer.py (13 lines)
Tested with
Python 3.12.15 | confluent-kafka==2.16.1
What this does not show
The broker is Redpanda, which speaks the Kafka protocol but is not Apache Kafka, so edge cases can differ. One broker, one laptop, ten made-up events: no replication, no failure test, no load. You need Docker to follow along.
Sources
Learning IT by Example.