Detect Data Drift: PSI and K-S by Hand in Python
Moving the mean up by 4 gave a PSI of 0.153; the K-S test flagged a shift of 1 that PSI scored as small.
Video #158 ยท MLOps and deployment
Steps in the video
- 0:00 Mean up 4: PSI 0.153. A shift of 1: K-S flags it, PSI barely moves.
- 0:32 The idea
- 0:54 Setup
- 1:06 The made-up data
- 1:26 Bins for P S I
- 1:52 The P S I sum
- 2:08 The K S statistic
- 2:28 When is D big?
- 4:03 The thresholds
- 4:59 What this does not show
- 5:25 Mistakes to avoid
- 5:47 My take and next steps
- 6:08 Recap
Files
- data.py (18 lines)
- ks.py (21 lines)
- psi.py (23 lines)
- run.py (26 lines)
Tested with
Python 3.12.15
What this does not show
Made-up order values for one feature from a bell curve; no model is run; the PSI and K-S thresholds quoted are rules of thumb, and the PSI score changes with the number of bins.
Sources
- https://arize.com/glossary/population-stability-index-psi/
- https://docs.evidentlyai.com/metrics/customize_data_drift
- https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html
- https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test
- https://scholarworks.wmich.edu/dissertations/3208/
Learning IT by Example.