learning-it by example

Back to the video

Data Contracts: Stop Breaking Downstream

In a made-up feed, a renamed column broke the dashboard; a contract check found 10 problems before any load.

Video #107 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 One renamed column broke the dashboard. A contract check found ten problems first.
  2. 0:31 The idea
  3. 1:06 Setup
  4. 1:20 The contract
  5. 1:52 Two versions of the feed
  6. 2:12 A loader with no contract
  7. 2:58 Checking each row
  8. 3:22 Unique columns
  9. 4:00 The producer proposes a new contract
  10. 4:14 Comparing two contracts
  11. 5:01 What this does not show
  12. 5:35 Mistakes to avoid
  13. 6:07 My take and next steps
  14. 6:25 Recap

Files

  • check.py (45 lines)
  • contract.py (28 lines)
  • run.py (55 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

The contract is a small Python dict that borrows ODCS key names, the feed is made up, and the check runs locally, not in a real CI server.

Sources

Learning IT by Example.