learning-it by example

#211 Data governance 5.9 min

Data Contract to Data Product: Validate and Diff in Python

Version 1.3.0 removed a column and changed a type. The check said it needs 2.0.0.

1. In the Python data product spec check, which two column changes are treated as breaking?
2. In the Python data product spec check, version 2 of the made-up customer_orders product was labelled 1.3.0. What version did the check say it needed?
3. In the Python data product spec check, what problem did validation find in version 2 of the made-up customer_orders spec?
4. How does the Python data product spec check read a column that has been renamed?
5. What can the Python data product spec check not see?
6. Why is the YAML file read with safe_load in the data product spec check?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Label version 2 correctly

    Copy the code folder first. In product_v2.yaml change version: 1.3.0 to version: 2.0.0 and run python run.py. You should see the too small line disappear, but the masking problem for customer_email stays.

    A hint

    The version label and the masking problem are two separate checks.

  2. Put the masking back

    In product_v2.yaml add masking: hash to the customer_email column, inside the curly braces. Run python run.py. You should see v2 1.3.0: valid, and the too small line still there. Remove the masking line again afterwards, so exercise 4 gives the numbers it states.

    A hint

    Valid and correctly numbered are two different things in this check.

  3. Make a safe update

    In product_v2.yaml change total_gbp back to type decimal and order_date back to placed_on. Run python run.py. You should see only safe added channel in the changes, and no too small line.

    A hint

    Only a removed column or a changed type counts as breaking here. An added column is safe.

  4. Break the description rule

    In product_v2.yaml change the channel description to empty quotes, like description: "". Predict how many problems v2 will report, then run python run.py. You should see 2 problems, the masking one and channel: no description.

    A hint

    The validate function wants every column to have a description.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up product in my own small format, not the Open Data Contract Standard; the check compares only column names and types, and it prints its findings without failing a build.

Up next in Data governance

More in Data governance

All Data governance videos

Related videos