learning-it by example

Back to the video

Incremental Loads: Watermarks and the Late Update Trap

Full reloads copied 3,011 rows over three loads; a watermark copied 1,010 and missed one late update.

Video #111 ยท Data engineering and pipelines

Steps in the video

  1. 0:00 Full reload copied 3011 rows. Incremental copied 1010, and missed one update.
  2. 0:28 The idea
  3. 1:01 Setup
  4. 1:27 A made-up source
  5. 2:07 Full reload
  6. 2:19 Where the last load stopped
  7. 2:37 The incremental load
  8. 3:10 The check
  9. 3:20 Run the full reload
  10. 4:38 What this does not show
  11. 5:07 Mistakes to avoid
  12. 5:55 My take and next steps
  13. 6:17 Recap

Files

  • load.py (42 lines)
  • run.py (43 lines)
  • source.py (39 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

One made-up table in SQLite with no deletes, no time zones and no timings; the one hour lookback is a choice, not a rule.

Sources

Learning IT by Example.