Medallion Architecture: Bronze, Silver, Gold in Python
A messy CSV summed straight from raw says 99 pounds; after bronze, silver and gold it is 86.
Video #263 ยท Data modelling
Steps in the video
- 0:00 A messy CSV summed straight from raw says 99. After three layers: 86.
- 0:27 The idea
- 1:02 Setup
- 1:34 Bronze: land it as text
- 1:56 Silver rules: dates and countries
- 2:30 Silver rules: amounts
- 2:52 Silver: keep one row per order
- 3:19 Gold: a table for the report
- 3:34 Run bronze
- 5:00 What this does not show
- 5:28 Mistakes to avoid
- 6:00 My take and next steps
- 6:22 Recap
Files
- layers.py (64 lines)
- orders_raw.csv (10 lines)
- run.py (62 lines)
Tested with
Python 3.12.15
What this does not show
Nine made-up rows in SQLite, not a lakehouse; no Delta tables, Spark or incremental loads, and the pattern is described by vendors.
Sources
- https://docs.databricks.com/aws/en/lakehouse/medallion
- https://docs.python.org/3/library/csv.html
- https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture
- https://www.databricks.com/glossary/medallion-architecture
- https://www.sqlite.org/datatype3.html
- https://www.sqlite.org/lang_expr.html#castexpr
Learning IT by Example.