learning-it by example

Back to the video

Detect Data Drift: PSI and K-S by Hand in Python

Moving the mean up by 4 gave a PSI of 0.153; the K-S test flagged a shift of 1 that PSI scored as small.

Video #158 ยท MLOps and deployment

Steps in the video

  1. 0:00 Mean up 4: PSI 0.153. A shift of 1: K-S flags it, PSI barely moves.
  2. 0:32 The idea
  3. 0:54 Setup
  4. 1:06 The made-up data
  5. 1:26 Bins for P S I
  6. 1:52 The P S I sum
  7. 2:08 The K S statistic
  8. 2:28 When is D big?
  9. 4:03 The thresholds
  10. 4:59 What this does not show
  11. 5:25 Mistakes to avoid
  12. 5:47 My take and next steps
  13. 6:08 Recap

Files

  • data.py (18 lines)
  • ks.py (21 lines)
  • psi.py (23 lines)
  • run.py (26 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Made-up order values for one feature from a bell curve; no model is run; the PSI and K-S thresholds quoted are rules of thumb, and the PSI score changes with the number of bins.

Sources

Learning IT by Example.