#115 Data engineering and pipelines 6.7 min
Schema Evolution Without Breaking Pipelines
Three changes to a made-up feed: a positional reader loaded all 8 rows without an error, and 4 were wrong.
Run the three readers
Copy the code folder first. Run python run.py positional, then python run.py by_name, then python run.py tolerant. For each, look at the list of orders at the end and count the rows where the amount is not a whole number of pence, or where the amount column holds text. Write down which reader skipped a feed and which one refused a feed.
A hint
Look at the stored type column on the right of each row, and the lines that start with v2, v3 and v4.
A plain whole number slips through
In feeds.py, change the v4 amounts from 12.50 and 8.00 to plain 12 and 8, then run python run.py tolerant. Predict first whether version 4 will be refused. On this machine it prints loaded 2 for v4, and orders 1007 and 1008 end up with 12 and 8 as whole pence.
A hint
The reader checks that the value is a whole number, not what the number means.
Teach the reader a new alias
In feeds.py, change the header in the v3 feed from client to buyer. Run python run.py tolerant and see the version 3 line. Then add buyer to ALIASES in readers.py, mapping it to customer, and run again. On this machine the first run prints REFUSED: missing required column customer for v3, and the second run prints loaded 2.
A hint
ALIASES maps a name the feed uses to the name in the table.
Make the pound feed pass on purpose
Change the v4 amounts in feeds.py to 1250 and 800, as if the supplier had converted them to pence themselves. Run python run.py tolerant. Predict first what happens to v4. On this machine it prints loaded 2. Then explain in one sentence why the reader accepts this feed and refused the original one.
A hint
Whole numbers can be converted to integers, and 12.50 cannot.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Eight made-up orders in SQLite; the tolerant reader refuses a unit change rather than fixing it, and no Avro, Delta Lake or schema registry is run.
Up next in Data engineering and pipelines
More in Data engineering and pipelines
- #13 Dagster Data Pipeline with Blocking Checks: No Server Needed
- #14 dbt and DuckDB: A Full Project, Tests, and a Bad Row
- #15 Kafka Streaming Basics in Python with a Redpanda Broker
- #107 Data Contracts: Stop Breaking Downstream
- #111 Incremental Loads: Watermarks and the Late Update Trap
- #112 Idempotent Loads: Append, Upsert, Replace, Delete Insert
Related videos
- #263 Medallion Architecture: Bronze, Silver, Gold in Python Data modelling
- #107 Data Contracts: Stop Breaking Downstream Data engineering and pipelines
- #111 Incremental Loads: Watermarks and the Late Update Trap Data engineering and pipelines
- #112 Idempotent Loads: Append, Upsert, Replace, Delete Insert Data engineering and pipelines