learning-it by example

Back to the video

Column-Level Lineage from SQL with SQLGlot

Which columns feed this number? SQLGlot can answer in 15 lines, and I will show you where it misleads.

Video #205 ยท Data governance

Steps in the video

  1. 0:00 Which columns feed this number?
  2. 0:31 The idea
  3. 0:58 Setup
  4. 1:13 The schema
  5. 1:24 The helper
  6. 1:50 Three queries
  7. 2:16 Run it
  8. 2:50 Impact analysis
  9. 3:26 Three traps
  10. 4:09 What this does not show
  11. 4:34 Mistakes to avoid
  12. 4:54 My take and next steps
  13. 5:16 Recap

Files

  • columns.py (17 lines)
  • gaps.py (14 lines)
  • impact.py (12 lines)
  • queries.py (20 lines)
  • show.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | sqlglot==30.23.0

What this does not show

Three simple queries in the default dialect; filter and join columns are not in this lineage, and it reads SQL text, not what ran.

Sources

Learning IT by Example.