#07 · SQL and analytics · 5.9 minSQL Window Functions and CTEs in DuckDB: Two Mistakes
Learn CTEs and window functions with DuckDB on a tiny orders table, then see two common mistakes, with real output.
Python 3.12.15 | duckdb==2.0.016 videos tagged "data engineering".
#07 · SQL and analytics · 5.9 minLearn CTEs and window functions with DuckDB on a tiny orders table, then see two common mistakes, with real output.
Python 3.12.15 | duckdb==2.0.0
#08 · Python for data · 6.1 minRun pandas, Polars and DuckDB on the same ten million rows on one laptop, time them carefully, and check that the answers agree.
Python 3.12.15 | pandas==3.1.0, polars==2.1.0, duckdb==2.0.0, pyarrow==26.0.0
#13 · Data engineering and pipelines · 5.8 minBuild a three-step data pipeline with Dagster in one file, with no server or Docker, and teach it to stop when the data is bad.
Python 3.12.15 | dagster==1.13.27, duckdb==2.0.0
#14 · Data engineering and pipelines · 6.2 minBuild a complete dbt project on DuckDB with models and tests, then watch a test catch a bad row that dbt run lets through.
Python 3.12.15 | dbt-core==1.12.6, dbt-duckdb==1.11.1, duckdb==2.0.0
#15 · Data engineering and pipelines · 5.7 minRun a Kafka-compatible broker in Docker, stream ten events from Python, read them back, and see why a consumer can show nothing.
Python 3.12.15 | confluent-kafka==2.16.1
#107 · Data engineering and pipelines · 6.9 minIn a made-up feed, a renamed column broke the dashboard; a contract check found 10 problems before any load.
Python 3.12.15
#111 · Data engineering and pipelines · 6.7 minFull reloads copied 3,011 rows over three loads; a watermark copied 1,010 and missed one late update.
Python 3.12.15
#112 · Data engineering and pipelines · 6.5 minLoading the same day of sales twice with a plain append doubled it from 63.35 to 126.70; three other strategies stayed at 6 rows.
Python 3.12.15On 50,000 made-up orders, a Parquet file was 52% of the CSV, and one query needed the amount column from 1 row group of 5.
Python 3.12.15 | pyarrow==26.0.0On 60,000 made-up orders, a one-month query opened 1 of 12 files by month, but 300 of 300 by customer.
Python 3.12.15 | pyarrow==26.0.0
#115 · Data engineering and pipelines · 6.7 minThree changes to a made-up feed: a positional reader loaded all 8 rows without an error, and 4 were wrong.
Python 3.12.15Mypy reported 4 errors from 3 bugs without running the code, and missed a wrong average.
Python 3.12.15 | mypy==2.4.0
#205 · Data governance · 5.7 minWhich columns feed this number? SQLGlot can answer in 15 lines, and I will show you where it misleads.
Python 3.12.15 | sqlglot==30.23.0
#216 · Data governance · 6.5 minSix made-up pipeline runs, three gaps: an evidence report built straight from the logs.
Python 3.12.15
#263 · Data modelling · 6.8 minA messy CSV summed straight from raw says 99 pounds; after bronze, silver and gold it is 86.
Python 3.12.15
#266 · Data modelling · 6.4 minTwo late clicks made an append-only job report 7 sessions; rebuilt from event time there were 4.
Python 3.12.15