Incremental Loads: Watermarks and the Late Update Trap
Full reloads copied 3,011 rows over three loads; a watermark copied 1,010 and missed one late update.
Video #111 ยท Data engineering and pipelines
Steps in the video
- 0:00 Full reload copied 3011 rows. Incremental copied 1010, and missed one update.
- 0:28 The idea
- 1:01 Setup
- 1:27 A made-up source
- 2:07 Full reload
- 2:19 Where the last load stopped
- 2:37 The incremental load
- 3:10 The check
- 3:20 Run the full reload
- 4:38 What this does not show
- 5:07 Mistakes to avoid
- 5:55 My take and next steps
- 6:17 Recap
Files
- load.py (42 lines)
- run.py (43 lines)
- source.py (39 lines)
Tested with
Python 3.12.15
What this does not show
One made-up table in SQLite with no deletes, no time zones and no timings; the one hour lookback is a choice, not a rule.
Sources
Learning IT by Example.