learning-it by example

Back to the video

Dagster Data Pipeline with Blocking Checks: No Server Needed

Build a three-step data pipeline with Dagster in one file, with no server or Docker, and teach it to stop when the data is bad.

Video #13 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 A pipeline that refuses bad data
  2. 0:17 Assets, not tasks
  3. 0:49 Setup
  4. 1:04 Two simple assets
  5. 1:25 A check that blocks
  6. 1:52 The last step
  7. 2:08 Run it
  8. 2:40 Feed it bad data
  9. 3:17 A common mistake
  10. 3:48 What this does not show
  11. 4:26 Mistakes to avoid
  12. 4:59 My take and next steps
  13. 5:25 Recap

Files

  • bad_data.py (14 lines)
  • pipeline.py (38 lines)
  • run.py (13 lines)
  • wrong_name.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | dagster==1.13.27, duckdb==2.0.0

What this does not show

One file, five made-up orders, one process: no schedules, retries, alerts, run history or web interface. The blocking check only stops the one problem it was written for (a negative amount). Shown for dagster 1.13.27.

Sources

Learning IT by Example.