learning-it by example

#115 Data engineering and pipelines 6.7 min

Schema Evolution Without Breaking Pipelines

Three changes to a made-up feed: a positional reader loaded all 8 rows without an error, and 4 were wrong.

1. A made-up supplier feed changes three times. A reader that takes the first three fields by position loads all 8 rows. How many of those rows are wrong?
2. What does a tolerant reader do when a feed sends a column that the target table does not have?
3. How did the tolerant reader treat the version 4 feed, where amounts switched from pence to pounds, like 12.50?
4. The tolerant reader caught the unit change only because of the decimal point. What does the video say about a pounds amount written as a plain 8?
5. How does the tolerant reader know which columns are required?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the three readers

    Copy the code folder first. Run python run.py positional, then python run.py by_name, then python run.py tolerant. For each, look at the list of orders at the end and count the rows where the amount is not a whole number of pence, or where the amount column holds text. Write down which reader skipped a feed and which one refused a feed.

    A hint

    Look at the stored type column on the right of each row, and the lines that start with v2, v3 and v4.

  2. A plain whole number slips through

    In feeds.py, change the v4 amounts from 12.50 and 8.00 to plain 12 and 8, then run python run.py tolerant. Predict first whether version 4 will be refused. On this machine it prints loaded 2 for v4, and orders 1007 and 1008 end up with 12 and 8 as whole pence.

    A hint

    The reader checks that the value is a whole number, not what the number means.

  3. Teach the reader a new alias

    In feeds.py, change the header in the v3 feed from client to buyer. Run python run.py tolerant and see the version 3 line. Then add buyer to ALIASES in readers.py, mapping it to customer, and run again. On this machine the first run prints REFUSED: missing required column customer for v3, and the second run prints loaded 2.

    A hint

    ALIASES maps a name the feed uses to the name in the table.

  4. Make the pound feed pass on purpose

    Change the v4 amounts in feeds.py to 1250 and 800, as if the supplier had converted them to pence themselves. Run python run.py tolerant. Predict first what happens to v4. On this machine it prints loaded 2. Then explain in one sentence why the reader accepts this feed and refused the original one.

    A hint

    Whole numbers can be converted to integers, and 12.50 cannot.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Eight made-up orders in SQLite; the tolerant reader refuses a unit change rather than fixing it, and no Avro, Delta Lake or schema registry is run.

Up next in Data engineering and pipelines

More in Data engineering and pipelines

All Data engineering and pipelines videos

Related videos