learning-it by example

Back to the video

Schema Evolution Without Breaking Pipelines

Three changes to a made-up feed: a positional reader loaded all 8 rows without an error, and 4 were wrong.

Video #115 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 Three feed changes. The positional reader loaded all 8 rows, and 4 are wrong.
  2. 0:33 The idea
  3. 1:08 Setup
  4. 1:21 The feeds and the table
  5. 1:52 Two fragile readers
  6. 2:11 The contract
  7. 2:38 The tolerant reader
  8. 3:02 Run the positional reader
  9. 3:59 Adding a column safely
  10. 4:48 What this does not show
  11. 5:16 Mistakes to avoid
  12. 5:54 My take and next steps
  13. 6:19 Recap

Files

  • feeds.py (13 lines)
  • readers.py (48 lines)
  • run.py (66 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Eight made-up orders in SQLite; the tolerant reader refuses a unit change rather than fixing it, and no Avro, Delta Lake or schema registry is run.

Sources

Learning IT by Example.