learning-it by example

Back to the video

Cross-Validation Done Right: The Leakage Mistake

On pure noise with random labels, cross-validation reported 0.89 accuracy, until one step moved.

Video #185 ยท Machine learning and data science

Steps in the video

  1. 0:00 Random labels, 0.89 cross-validated accuracy
  2. 0:30 The idea
  3. 1:02 Setup
  4. 1:26 Two made-up datasets
  5. 1:52 One split, five times
  6. 2:55 The leakage mistake
  7. 4:05 What this does not show
  8. 4:36 Mistakes to avoid
  9. 5:11 My take and next steps
  10. 5:43 Recap

Files

  • data.py (18 lines)
  • run.py (39 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2

What this does not show

Small made-up datasets, one simple model, no tuning, and five folds give only a rough estimate.

Sources

Learning IT by Example.