learning-it by example

Back to the video

Data Contract to Data Product: Validate and Diff in Python

Version 1.3.0 removed a column and changed a type. The check said it needs 2.0.0.

Video #211 ยท Data governance

Steps in the video

  1. 0:00 A data product spec in YAML, checked by Python, caught a breaking change labelled as minor
  2. 0:36 The idea
  3. 1:15 Setup
  4. 1:36 Version one
  5. 2:04 Version two
  6. 2:30 Validate
  7. 2:52 Compare and bump
  8. 3:20 Run both
  9. 4:07 What this does not show
  10. 4:37 Mistakes to avoid
  11. 5:01 My take and next steps
  12. 5:27 Recap

Files

  • product_v1.yaml (10 lines)
  • product_v2.yaml (11 lines)
  • run.py (16 lines)
  • validate.py (40 lines)

Download the code (zip)

Tested with

Python 3.12.15 | PyYAML==6.0.3

What this does not show

Made-up product in my own small format, not the Open Data Contract Standard; the check compares only column names and types, and it prints its findings without failing a build.

Sources

Learning IT by Example.