#186 Machine learning and data science 6.2 min
Data Leakage: The Bug That Fakes Accuracy, Reproduced
One column filled in after delivery lifted test accuracy from 0.69 to 0.95.
Run both leaks and compare
Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py target and then python run.py groups. In the video's run, adding refund_paid lifted accuracy from 0.69 to 0.95, and random rows scored 0.98 while whole patients scored 0.57. Write one sentence on why each jump is not real.
A hint
One leak is a column filled in after the outcome. The other is the same patient on both sides of the split.
List when each column is known
Open data.py and read make_deliveries. For each of the three columns, distance, rush_hour and refund_paid, write down when it gets its value, before the delivery or after. Then say which column you would drop before training a model that predicts lateness at the moment an order is placed.
A hint
The refund is decided only once the delivery has happened.
Predict a weaker leak
In data.py, change refund_if_not = rng.uniform(0, 1, n) < 0.03 to < 0.5, so that on time deliveries often get a refund too. Predict what happens to the line refund_paid alone as a guess, and to the accuracy of the model with the leaky column, then run python run.py target and compare.
A hint
The column is only a useful leak if it differs between late and on time deliveries.
One scan per patient
In data.py, change scans_each=3 to scans_each=1. Predict what happens to the gap between the random rows line and the whole patients line, then run python run.py groups and compare. Explain in one sentence why the gap should change.
A hint
With one scan per patient, no patient can appear on both sides of a split.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up data with leaks built on purpose; small test sets; time leaks and preprocessing leaks are not shown.
Up next in Machine learning and data science
More in Machine learning and data science
Related videos
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data Machine learning and data science
- #193 Clustering: k-means and When It Fails, in Python Machine learning and data science
- #183 Precision, Recall, F1: Which Metric Matters? Machine learning and data science
