#184 Machine learning and data science 6 min
Overfitting You Can See: Train vs Validation Error
Degree 14 had a training error of 0.00 and a validation error of 3.27.
Read the table of errors
Copy the code folder first. Run python run.py. Find the degree where training error and validation error are furthest apart, and the degree with the lowest validation error. When I ran it, the lowest was degree 5, and degree 14 had training error 0.00 and validation error 3.27.
A hint
numpy is the only package this needs.
Try a different list of degrees
In run.py, change DEGREES to [1, 5, 13, 14]. Predict whether degree 13 looks closer to degree 5 or to degree 14, then run it. In my check, degree 13 had training error 0.04 and validation error 0.56, and the lowest validation error was still degree 5. Then put DEGREES back to [1, 2, 3, 4, 5, 6, 8, 10, 12, 14], because later exercises assume it.
A hint
With 15 points, a degree 14 polynomial can pass through every point.
Turn the noise down
In data.py, change noise=0.15 to noise=0.05. Predict whether overfitting at degree 14 gets better or worse, then run python run.py. In my check, degree 14 still overfit, with validation error 1.09, and the lowest validation error was still degree 5.
A hint
Compare the degree 14 row with the one in the original run.
Use a different random seed
Put the noise back to 0.15. In data.py, change seed=5 to seed=6. Predict whether the lowest validation error is still at degree 5, then run python run.py. In my check, the lowest was degree 6 and degree 14 had validation error 6.80. This is the video's point that one validation set gives one estimate.
A hint
The seed changes both the validation points and the noise.
Give the model more training points
Put the seed back to 5. In data.py, change n_train=15 to n_train=30. Predict what happens to degree 14, then run python run.py. In my check, degree 14 no longer blew up, and the lowest validation error moved to degree 12.
A hint
A degree 14 curve can no longer pass through every one of 30 points.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up points from a known curve, one train and validation split, and no separate test set.
Up next in Machine learning and data science
More in Machine learning and data science
Related videos
- #296 Attention in 40 Lines of NumPy: The Core of Transformers LLM engineering and evals
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #194 Anomaly Detection Basics: z-Score, IQR, Isolation Forest Machine learning and data science
- #242 Architecture Decision Records for AI Choices, With a Lint Enterprise architecture and AI agents
