learning-it by example

#158 MLOps and deployment 6.5 min

Detect Data Drift: PSI and K-S by Hand in Python

Moving the mean up by 4 gave a PSI of 0.153; the K-S test flagged a shift of 1 that PSI scored as small.

1. How do PSI and the Kolmogorov-Smirnov statistic each compare a reference sample with a current sample?
2. In a hand-written Python PSI check on 5,000 made-up order values, moving the mean up by 4 gave what PSI?
3. In the video, a mean shift of 1 with 50,000 rows had a PSI of 0.012 but a K-S D of 0.051 over its limit of 0.020. What does that show?
4. How does the video's PSI code choose its bins?
5. The video scored the same mean +4 shift with 5, 10, 20 and 50 bins. What did it find?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Check the K-S statistic by hand

    Copy the code folder first. In run.py in the check mode, add a print of ks_statistic([1, 2, 3], [2, 3, 4]). Work out the biggest gap between the two cumulative shares by hand first, then run python run.py check. On my machine it printed about 0.333.

    A hint

    At each value, compare the share of each list seen so far and keep the largest difference.

  2. A bigger sample gives a smaller limit

    In run.py in the check mode, add prints of critical(100, 100) and critical(5000, 5000). Predict which limit is larger, then run python run.py check. On my machine the first printed about 0.192 and the second about 0.027.

    A hint

    The limit has the square root of (n plus m) over (n times m) in it.

  3. Try more bin counts

    In run.py in the bins mode change (5, 10, 20, 50) to (2, 10, 100, 200). Predict the trend, then run python run.py bins. On my machine 2 bins gave 0.100, 10 bins still gave 0.153, 100 bins gave 0.195 and 200 bins gave 0.234.

    A hint

    Compare with the video, where 5 bins gave 0.144 and 50 bins gave 0.173.

  4. Move the mean up by 2

    In data.py replace "mean +1": sample(41, 5000) with "mean +2": sample(42, 5000). Run python run.py table and read the last row. On my machine it showed a PSI of 0.045, a D of 0.094 and a limit of 0.027. Compare this with the mean +1 row from the video, which had a PSI of 0.013 and a D of 0.056, and say which score moved more.

    A hint

    Look at how each score changed relative to its starting value.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up order values for one feature from a bell curve; no model is run; the PSI and K-S thresholds quoted are rules of thumb, and the PSI score changes with the number of bins.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos