learning-it by example

#186 Machine learning and data science 6.2 min

Data Leakage: The Bug That Fakes Accuracy, Reproduced

One column filled in after delivery lifted test accuracy from 0.69 to 0.95.

1. How do the scikit-learn docs, as the video quotes them, define data leakage?
2. On the 600 made-up deliveries, what test accuracy did the decision tree get without the refund_paid column, and with it?
3. What quick check does the video use to expose a column that leaks the answer?
4. In the patient scans example, what did the random-rows cross-validation score, compared with the whole-patients split?
5. The honest patient score was 0.57. What does the video say about how good that is?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run both leaks and compare

    Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py target and then python run.py groups. In the video's run, adding refund_paid lifted accuracy from 0.69 to 0.95, and random rows scored 0.98 while whole patients scored 0.57. Write one sentence on why each jump is not real.

    A hint

    One leak is a column filled in after the outcome. The other is the same patient on both sides of the split.

  2. List when each column is known

    Open data.py and read make_deliveries. For each of the three columns, distance, rush_hour and refund_paid, write down when it gets its value, before the delivery or after. Then say which column you would drop before training a model that predicts lateness at the moment an order is placed.

    A hint

    The refund is decided only once the delivery has happened.

  3. Predict a weaker leak

    In data.py, change refund_if_not = rng.uniform(0, 1, n) < 0.03 to < 0.5, so that on time deliveries often get a refund too. Predict what happens to the line refund_paid alone as a guess, and to the accuracy of the model with the leaky column, then run python run.py target and compare.

    A hint

    The column is only a useful leak if it differs between late and on time deliveries.

  4. One scan per patient

    In data.py, change scans_each=3 to scans_each=1. Predict what happens to the gap between the random rows line and the whole patients line, then run python run.py groups and compare. Explain in one sentence why the gap should change.

    A hint

    With one scan per patient, no patient can appear on both sides of a split.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up data with leaks built on purpose; small test sets; time leaks and preprocessing leaks are not shown.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos