learning-it by example

#13 Data engineering and pipelines 5.8 min

Dagster Data Pipeline with Blocking Checks: No Server Needed

Build a three-step data pipeline with Dagster in one file, with no server or Docker, and teach it to stop when the data is bad.

1. In Dagster, how does the pipeline know that clean_orders must run after raw_orders?
2. In the Dagster order pipeline's normal run, how many clean rows were left from the five raw orders?
3. In the Dagster pipeline, after an order of minus forty was added, which assets were built?
4. What does the video say the blocking check does not protect you from?
5. An asset took an argument named raw_order, but the upstream asset is raw_orders. What happened?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fix the typo

    Copy the code folder first. In wrong_name.py change raw_order to raw_orders in both places. Change the materialize line to keep its result, and print result.output_for_node('order_count'). Predict the number it prints, then run python wrong_name.py. This needs dagster and duckdb at the versions in requirements.txt.

    A hint

    order_count returns the length of raw_orders, which is a list of orders.

  2. Fill the gap in the data

    In pipeline.py change the order for Ava on 2026-01-11 from None to 25.0. Predict the clean rows count and the three daily totals that run.py prints, then run it.

    A hint

    clean_orders only drops an order when its amount is missing.

  3. Try an amount of zero

    In bad_data.py change the extra order's amount from -40.0 to 0.0. The check fails only for amounts below zero. Predict the run succeeded line, the check line and the built list, then run python bad_data.py.

    A hint

    Look at the test row[2] < 0 in the check.

  4. Add a second blocking check

    In pipeline.py add a second blocking asset check on clean_orders named no_huge_amounts that fails if any amount is over 150, and add it to EVERYTHING. In bad_data.py change -40.0 to 20.0, so the first check passes. Predict what python bad_data.py prints, then run it.

    A hint

    Copy the shape of no_negative_amounts: an asset_check decorator with asset=clean_orders and blocking=True.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One file, five made-up orders, one process: no schedules, retries, alerts, run history or web interface. The blocking check only stops the one problem it was written for (a negative amount). Shown for dagster 1.13.27.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos