#192 Machine learning and data science 6.2 min
Time Series Forecasting: Score the Baselines First
Yesterday as the forecast missed by 12.3 a day; the same weekday last week missed by 6.8, and a model only got to 5.4.
Run the baselines and the models
Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py. Rank the six lines from the smallest mean miss to the largest. In the video's run, the order was noise floor 4.8, linear regression 5.4, the 4 week mean 5.6, gradient boosting 5.8, seasonal naive 6.8 and naive 12.3.
A hint
The noise floor line uses the true rule without noise, so no real forecast should beat it.
Average over more weeks
In forecasts.py, change weeks=4 in seasonal_mean to weeks=8. Predict whether the 8 week average beats the 4 week average, then run python run.py and compare. In my check of the baselines, the mean miss went up from 5.6 to 6.0.
A hint
Averaging more weeks smooths the noise but uses older weeks, and the sales trend upward slowly. On one 8 week test window a gap of a few tenths can go either way.
Turn the noise down
In data.py, change rng.normal(0, 6, days) to rng.normal(0, 2, days). Predict what happens to the naive forecast, the seasonal naive forecast and the noise floor, then run python run.py. In my check of the baselines, the linear model and the noise floor, naive missed by 11.4, seasonal naive by 2.3, the 4 week mean by 2.1, the linear model by 1.9 and the noise floor by 1.6. I did not check gradient boosting.
A hint
Less noise helps the methods that use the same weekday more than the one that uses yesterday.
Try a shorter test window
In run.py, change both 56 values (in test and in train) to 28. Predict whether the order of the baselines changes, then run python run.py. In my check of the baselines, the linear model and the noise floor, naive missed by 12.6, seasonal naive by 6.9, the 4 week mean by 6.3, the linear model by 6.3 and the noise floor by 5.0. The video says one test window may not settle small gaps.
A hint
The video warns that gaps of a few tenths could change order on another window.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Made-up, clean weekly data; one-day-ahead forecasts over one eight-week window; gradient boosting untuned; no classical models.
Up next in Machine learning and data science
More in Machine learning and data science
- #182 Linear Regression From Scratch, Then scikit-learn
- #183 Precision, Recall, F1: Which Metric Matters?
- #184 Overfitting You Can See: Train vs Validation Error
- #185 Cross-Validation Done Right: The Leakage Mistake
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data
Related videos
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced Machine learning and data science
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data Machine learning and data science
- #193 Clustering: k-means and When It Fails, in Python Machine learning and data science
