Cross-Validation Done Right: The Leakage Mistake
On pure noise with random labels, cross-validation reported 0.89 accuracy, until one step moved.
Video #185 ยท Machine learning and data science
Steps in the video
Files
- data.py (18 lines)
- run.py (39 lines)
Tested with
Python 3.12.15 | numpy==2.5.4, scikit-learn==1.9.1, scipy==1.18.1, joblib==1.6.0, narwhals==2.27.1, threadpoolctl==3.7.0, cloudpickle==3.1.2
What this does not show
Small made-up datasets, one simple model, no tuning, and five folds give only a rough estimate.
Sources
- https://pypi.org/pypi/numpy/json
- https://pypi.org/pypi/scikit-learn/json
- https://scikit-learn.org/stable/common_pitfalls.html
- https://scikit-learn.org/stable/modules/cross_validation.html
- https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html
- https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html
- https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
Learning IT by Example.