#185 Machine learning and data science 6.3 min
Cross-Validation Done Right: The Leakage Mistake
On pure noise with random labels, cross-validation reported 0.89 accuracy, until one step moved.
Run both parts and read the folds
Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py split and then python run.py leak. For each printed line, read the folds, the mean and the sd. In the video's run, the split seeds gave 0.60 to 0.90, the leak line gave a mean of 0.89 and the pipeline line gave 0.52.
A hint
The leak part uses pure noise, so an honest score is about 0.5.
Predict five more splits
In run.py, change range(5) in the split function to range(5, 10). Predict whether the five test accuracies will all be close to 0.77 or spread out like the first five, then run python run.py split and compare. Say what this tells you about trusting one split.
A hint
Only the seed that decides which rows land in the test set changes.
Predict a smaller selection
In the leak function, change k=20 to k=2 in both places. Before running python run.py leak, predict which line, the wrong way or the pipeline, will change more, and which will stay near a coin flip. Then run it and compare. Explain your result in one sentence using the word leakage.
A hint
The right way picks columns from training rows only, inside each fold.
Check a signal dataset with the pipeline
In the split function, build a pipeline with make_pipeline(SelectKBest(k=2), GaussianNB()) and score it with cross_val_score on the signal data X, y. Predict whether the mean will be close to the 0.77 from the plain model, then run it. Say why putting the selection inside the pipeline is the right way to do it.
A hint
Only the first two columns carry any signal in the made-up data.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Small made-up datasets, one simple model, no tuning, and five folds give only a rough estimate.
Up next in Machine learning and data science
More in Machine learning and data science
Related videos
- #316 Build an Eval Set From Customer Tickets Without Leakage Forward deployed engineer interviews
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced Machine learning and data science
- #159 Feature Stores: When a Table and a Join Are Enough MLOps and deployment
- #296 Attention in 40 Lines of NumPy: The Core of Transformers LLM engineering and evals
