Schema Evolution Without Breaking Pipelines
Three changes to a made-up feed: a positional reader loaded all 8 rows without an error, and 4 were wrong.
Video #115 ยท Data engineering and pipelines
Steps in the video
- 0:00 Three feed changes. The positional reader loaded all 8 rows, and 4 are wrong.
- 0:33 The idea
- 1:08 Setup
- 1:21 The feeds and the table
- 1:52 Two fragile readers
- 2:11 The contract
- 2:38 The tolerant reader
- 3:02 Run the positional reader
- 3:59 Adding a column safely
- 4:48 What this does not show
- 5:16 Mistakes to avoid
- 5:54 My take and next steps
- 6:19 Recap
Files
- feeds.py (13 lines)
- readers.py (48 lines)
- run.py (66 lines)
Tested with
Python 3.12.15
What this does not show
Eight made-up orders in SQLite; the tolerant reader refuses a unit change rather than fixing it, and no Avro, Delta Lake or schema registry is run.
Sources
- https://avro.apache.org/docs/1.12.0/specification/
- https://docs.databricks.com/aws/en/delta/update-schema
- https://docs.python.org/3/library/csv.html
- https://en.wikipedia.org/wiki/Schema_evolution
- https://martinfowler.com/bliki/TolerantReader.html
- https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html
- https://www.sqlite.org/datatype3.html
- https://www.sqlite.org/lang_altertable.html
- https://www.sqlite.org/stricttables.html
Learning IT by Example.