Data Contracts: Stop Breaking Downstream
In a made-up feed, a renamed column broke the dashboard; a contract check found 10 problems before any load.
Video #107 ยท Data engineering and pipelines
Steps in the video
- 0:00 One renamed column broke the dashboard. A contract check found ten problems first.
- 0:31 The idea
- 1:06 Setup
- 1:20 The contract
- 1:52 Two versions of the feed
- 2:12 A loader with no contract
- 2:58 Checking each row
- 3:22 Unique columns
- 4:00 The producer proposes a new contract
- 4:14 Comparing two contracts
- 5:01 What this does not show
- 5:35 Mistakes to avoid
- 6:07 My take and next steps
- 6:25 Recap
Files
- check.py (45 lines)
- contract.py (28 lines)
- run.py (55 lines)
Tested with
Python 3.12.15
What this does not show
The contract is a small Python dict that borrows ODCS key names, the feed is made up, and the check runs locally, not in a real CI server.
Sources
- https://bitol-io.github.io/open-data-contract-standard/latest/
- https://bitol-io.github.io/open-data-contract-standard/latest/fundamentals/
- https://bitol-io.github.io/open-data-contract-standard/latest/schema/
- https://docs.getdbt.com/docs/mesh/govern/model-contracts
- https://docs.python.org/3.12/library/sqlite3.html
- https://semver.org/
Learning IT by Example.