#107 Data engineering and pipelines 6.9 min
Data Contracts: Stop Breaking Downstream
In a made-up feed, a renamed column broke the dashboard; a contract check found 10 problems before any load.
Version the change as 2.0.0
Copy the code folder first. In contract.py change the version of PROPOSED from 1.1.0 to 2.0.0, then run python run.py diff. Predict whether the FAIL line still appears. When I ran it, the breaking changes were still listed but there was no FAIL line and the exit code was 0.
A hint
Compare the major number of the proposal with the major number of the old contract.
Fix the repeated order id
In contract.py change the third row of FEED_V2 so its order_id is 3 instead of 2, then run python run.py check. Predict how many problems feed v2 has now. When I ran it the answer was 9, because the repeated order id no longer counts.
A hint
Each row still has three other problems.
Make coupon required
In contract.py change the coupon column of CONTRACT from required False to required True, then run python run.py check. Predict how many problems feed v1 has, and which rows. When I ran it feed v1 had 2 problems, for the rows with no coupon.
A hint
Two of the three orders in FEED_V1 have coupon set to None.
Break feed version 1 and load it
In contract.py change the amount of the first row of FEED_V1 from 12.5 to None. Run python run.py check and then python run.py load. Predict what each says. When I ran it, check reported 1 problem for feed v1 because amount is required but missing, while the loader still showed 3 orders with revenue 37.5. That is why the check comes first.
A hint
The loader has no contract and takes whatever the feed holds.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The contract is a small Python dict that borrows ODCS key names, the feed is made up, and the check runs locally, not in a real CI server.
