learning-it by example

#185 Machine learning and data science 6.3 min

Cross-Validation Done Right: The Leakage Mistake

On pure noise with random labels, cross-validation reported 0.89 accuracy, until one step moved.

1. Why is scoring a model on one train and test split not enough?
2. On the made-up 100-row dataset with a weak signal, what did five-fold cross-validation report?
3. On pure noise with random labels, what is wrong with picking the best 20 columns on all rows and then cross-validating?
4. When the column selection and the model were put in one pipeline and that pipeline was cross-validated on the pure-noise data, what was the mean score?
5. The video gives a tip for when a score looks too good to be true. What is it?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run both parts and read the folds

    Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py split and then python run.py leak. For each printed line, read the folds, the mean and the sd. In the video's run, the split seeds gave 0.60 to 0.90, the leak line gave a mean of 0.89 and the pipeline line gave 0.52.

    A hint

    The leak part uses pure noise, so an honest score is about 0.5.

  2. Predict five more splits

    In run.py, change range(5) in the split function to range(5, 10). Predict whether the five test accuracies will all be close to 0.77 or spread out like the first five, then run python run.py split and compare. Say what this tells you about trusting one split.

    A hint

    Only the seed that decides which rows land in the test set changes.

  3. Predict a smaller selection

    In the leak function, change k=20 to k=2 in both places. Before running python run.py leak, predict which line, the wrong way or the pipeline, will change more, and which will stay near a coin flip. Then run it and compare. Explain your result in one sentence using the word leakage.

    A hint

    The right way picks columns from training rows only, inside each fold.

  4. Check a signal dataset with the pipeline

    In the split function, build a pipeline with make_pipeline(SelectKBest(k=2), GaussianNB()) and score it with cross_val_score on the signal data X, y. Predict whether the mean will be close to the 0.77 from the plain model, then run it. Say why putting the selection inside the pipeline is the right way to do it.

    A hint

    Only the first two columns carry any signal in the made-up data.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Small made-up datasets, one simple model, no tuning, and five folds give only a rough estimate.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos