Dagster Data Pipeline with Blocking Checks: No Server Needed
Build a three-step data pipeline with Dagster in one file, with no server or Docker, and teach it to stop when the data is bad.
Video #13 ยท Data engineering and pipelines
Steps in the video
Files
- bad_data.py (14 lines)
- pipeline.py (38 lines)
- run.py (13 lines)
- wrong_name.py (16 lines)
Tested with
Python 3.12.15 | dagster==1.13.27, duckdb==2.0.0
What this does not show
One file, five made-up orders, one process: no schedules, retries, alerts, run history or web interface. The blocking check only stops the one problem it was written for (a negative amount). Shown for dagster 1.13.27.
Sources
Learning IT by Example.