#211 Data governance 5.9 min
Data Contract to Data Product: Validate and Diff in Python
Version 1.3.0 removed a column and changed a type. The check said it needs 2.0.0.
Label version 2 correctly
Copy the code folder first. In product_v2.yaml change version: 1.3.0 to version: 2.0.0 and run python run.py. You should see the too small line disappear, but the masking problem for customer_email stays.
A hint
The version label and the masking problem are two separate checks.
Put the masking back
In product_v2.yaml add masking: hash to the customer_email column, inside the curly braces. Run python run.py. You should see v2 1.3.0: valid, and the too small line still there. Remove the masking line again afterwards, so exercise 4 gives the numbers it states.
A hint
Valid and correctly numbered are two different things in this check.
Make a safe update
In product_v2.yaml change total_gbp back to type decimal and order_date back to placed_on. Run python run.py. You should see only safe added channel in the changes, and no too small line.
A hint
Only a removed column or a changed type counts as breaking here. An added column is safe.
Break the description rule
In product_v2.yaml change the channel description to empty quotes, like description: "". Predict how many problems v2 will report, then run python run.py. You should see 2 problems, the masking one and channel: no description.
A hint
The validate function wants every column to have a description.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up product in my own small format, not the Open Data Contract Standard; the check compares only column names and types, and it prints its findings without failing a build.
Up next in Data governance
More in Data governance
- #203 A Business Glossary in YAML With Four Python Checks
- #204 Tag PII Columns in Python, Including a False Positive
- #205 Column-Level Lineage from SQL with SQLGlot
- #208 Right to Erasure Across a Pipeline: Delete, Scan, Restore
- #209 UK GDPR Data Map as Code: A Practical Checklist
- #210 A Data Quality Scorecard That Shows a Hidden Failure
