learning-it by example

#107 Data engineering and pipelines 6.9 min

Data Contracts: Stop Breaking Downstream

In a made-up feed, a renamed column broke the dashboard; a contract check found 10 problems before any load.

1. What is a data contract, as the video describes it?
2. Feed version 2 was checked against the contract before any load. How many problems did the check find in three rows, and what were they?
3. Why did the contract comparison fail the proposed version 1.1.0?
4. What kind of problem would still pass the contract check?
5. When a column is renamed, how does the contract comparison report it?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Version the change as 2.0.0

    Copy the code folder first. In contract.py change the version of PROPOSED from 1.1.0 to 2.0.0, then run python run.py diff. Predict whether the FAIL line still appears. When I ran it, the breaking changes were still listed but there was no FAIL line and the exit code was 0.

    A hint

    Compare the major number of the proposal with the major number of the old contract.

  2. Fix the repeated order id

    In contract.py change the third row of FEED_V2 so its order_id is 3 instead of 2, then run python run.py check. Predict how many problems feed v2 has now. When I ran it the answer was 9, because the repeated order id no longer counts.

    A hint

    Each row still has three other problems.

  3. Make coupon required

    In contract.py change the coupon column of CONTRACT from required False to required True, then run python run.py check. Predict how many problems feed v1 has, and which rows. When I ran it feed v1 had 2 problems, for the rows with no coupon.

    A hint

    Two of the three orders in FEED_V1 have coupon set to None.

  4. Break feed version 1 and load it

    In contract.py change the amount of the first row of FEED_V1 from 12.5 to None. Run python run.py check and then python run.py load. Predict what each says. When I ran it, check reported 1 problem for feed v1 because amount is required but missing, while the loader still showed 3 orders with revenue 37.5. That is why the check comes first.

    A hint

    The loader has no contract and takes whatever the feed holds.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

The contract is a small Python dict that borrows ODCS key names, the feed is made up, and the check runs locally, not in a real CI server.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos