learning-it by example

#184 Machine learning and data science 6 min

Overfitting You Can See: Train vs Validation Error

Degree 14 had a training error of 0.00 and a validation error of 3.27.

1. What is overfitting?
2. In the polynomial run on 15 made-up training points, what were the training and validation errors for the degree 14 curve?
3. According to Google's crash course as the video quotes it, what pattern in loss curves should make you suspect overfitting?
4. In the polynomial overfitting run on 15 made-up points, the lowest validation error was at degree 5. Why should you not read much into that?
5. The degree was picked using the validation score. What does the video say about reporting that score as the final result?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the table of errors

    Copy the code folder first. Run python run.py. Find the degree where training error and validation error are furthest apart, and the degree with the lowest validation error. When I ran it, the lowest was degree 5, and degree 14 had training error 0.00 and validation error 3.27.

    A hint

    numpy is the only package this needs.

  2. Try a different list of degrees

    In run.py, change DEGREES to [1, 5, 13, 14]. Predict whether degree 13 looks closer to degree 5 or to degree 14, then run it. In my check, degree 13 had training error 0.04 and validation error 0.56, and the lowest validation error was still degree 5. Then put DEGREES back to [1, 2, 3, 4, 5, 6, 8, 10, 12, 14], because later exercises assume it.

    A hint

    With 15 points, a degree 14 polynomial can pass through every point.

  3. Turn the noise down

    In data.py, change noise=0.15 to noise=0.05. Predict whether overfitting at degree 14 gets better or worse, then run python run.py. In my check, degree 14 still overfit, with validation error 1.09, and the lowest validation error was still degree 5.

    A hint

    Compare the degree 14 row with the one in the original run.

  4. Use a different random seed

    Put the noise back to 0.15. In data.py, change seed=5 to seed=6. Predict whether the lowest validation error is still at degree 5, then run python run.py. In my check, the lowest was degree 6 and degree 14 had validation error 6.80. This is the video's point that one validation set gives one estimate.

    A hint

    The seed changes both the validation points and the noise.

  5. Give the model more training points

    Put the seed back to 5. In data.py, change n_train=15 to n_train=30. Predict what happens to degree 14, then run python run.py. In my check, degree 14 no longer blew up, and the lowest validation error moved to degree 12.

    A hint

    A degree 14 curve can no longer pass through every one of 30 points.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up points from a known curve, one train and validation split, and no separate test set.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos