#158 MLOps and deployment 6.5 min
Detect Data Drift: PSI and K-S by Hand in Python
Moving the mean up by 4 gave a PSI of 0.153; the K-S test flagged a shift of 1 that PSI scored as small.
Check the K-S statistic by hand
Copy the code folder first. In run.py in the check mode, add a print of ks_statistic([1, 2, 3], [2, 3, 4]). Work out the biggest gap between the two cumulative shares by hand first, then run python run.py check. On my machine it printed about 0.333.
A hint
At each value, compare the share of each list seen so far and keep the largest difference.
A bigger sample gives a smaller limit
In run.py in the check mode, add prints of critical(100, 100) and critical(5000, 5000). Predict which limit is larger, then run python run.py check. On my machine the first printed about 0.192 and the second about 0.027.
A hint
The limit has the square root of (n plus m) over (n times m) in it.
Try more bin counts
In run.py in the bins mode change (5, 10, 20, 50) to (2, 10, 100, 200). Predict the trend, then run python run.py bins. On my machine 2 bins gave 0.100, 10 bins still gave 0.153, 100 bins gave 0.195 and 200 bins gave 0.234.
A hint
Compare with the video, where 5 bins gave 0.144 and 50 bins gave 0.173.
Move the mean up by 2
In data.py replace "mean +1": sample(41, 5000) with "mean +2": sample(42, 5000). Run python run.py table and read the last row. On my machine it showed a PSI of 0.045, a D of 0.094 and a limit of 0.027. Compare this with the mean +1 row from the video, which had a PSI of 0.013 and a D of 0.056, and say which score moved more.
A hint
Look at how each score changed relative to its starting value.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up order values for one feature from a bell curve; no model is run; the PSI and K-S thresholds quoted are rules of thumb, and the PSI score changes with the number of bins.
Up next in MLOps and deployment
More in MLOps and deployment
Related videos
- #157 Model Registry by Hand: Versions, Promote, Roll Back MLOps and deployment
- #159 Feature Stores: When a Table and a Join Are Enough MLOps and deployment
- #163 Secrets in ML Apps: Scan, Move Out, Rotate (Python) MLOps and deployment
- #164 Batch vs Real-Time Inference: A Latency Simulation MLOps and deployment