learning-it by example

Back to the video

Data Leakage: The Bug That Fakes Accuracy, Reproduced

One column filled in after delivery lifted test accuracy from 0.69 to 0.95.

Video #186 ยท Machine learning and data science

Steps in the video

  1. 0:00 Test accuracy 0.69, then 0.95 from one late column
  2. 0:28 The idea
  3. 1:03 Setup
  4. 1:26 A column from the future
  5. 1:49 Train it twice
  6. 2:50 The same patient on both sides
  7. 3:13 Split rows, or split patients
  8. 4:06 What this does not show
  9. 4:47 Mistakes to avoid
  10. 5:14 My take and next steps
  11. 5:45 Recap

Files

  • data.py (28 lines)
  • run.py (39 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2

What this does not show

Made-up data with leaks built on purpose; small test sets; time leaks and preprocessing leaks are not shown.

Sources

Learning IT by Example.