Profile Messy Customer Data in SQL Before Any Model
My data profile said 275 blank amounts. The file had 363. Here is why.
Video #309 ยท Forward deployed engineer interviews
Steps in the video
Files
- clean.py (18 lines)
- make_data.py (29 lines)
- run.py (26 lines)
- survey.py (35 lines)
Tested with
Python 3.12.15 | duckdb==2.0.0
What this does not show
A generated 5,000 row file from one fixed seed, with each problem put in on purpose. The clean-up keeps negative amounts, leaves 341 amounts missing, and keeps the latest row per email, a rule a customer must agree. The first, wrong count of 275 was a side check.
Sources
Learning IT by Example.