learning-it by example

#192 Machine learning and data science 6.2 min

Time Series Forecasting: Score the Baselines First

Yesterday as the forecast missed by 12.3 a day; the same weekday last week missed by 6.8, and a model only got to 5.4.

1. In forecasting, what does the seasonal naive method use as the forecast for a day?
2. On the made-up daily sales, how far did the naive forecast (yesterday) and the seasonal naive forecast (same weekday last week) miss on average?
3. Gradient boosting on 14 lags was compared with the plain mean of the last 4 same weekdays. What happened?
4. Why does the run hold back the last 56 days as the test set instead of shuffling days at random?
5. Why does the video say a model's error, such as 5.4, should always be shown next to a baseline?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the baselines and the models

    Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py. Rank the six lines from the smallest mean miss to the largest. In the video's run, the order was noise floor 4.8, linear regression 5.4, the 4 week mean 5.6, gradient boosting 5.8, seasonal naive 6.8 and naive 12.3.

    A hint

    The noise floor line uses the true rule without noise, so no real forecast should beat it.

  2. Average over more weeks

    In forecasts.py, change weeks=4 in seasonal_mean to weeks=8. Predict whether the 8 week average beats the 4 week average, then run python run.py and compare. In my check of the baselines, the mean miss went up from 5.6 to 6.0.

    A hint

    Averaging more weeks smooths the noise but uses older weeks, and the sales trend upward slowly. On one 8 week test window a gap of a few tenths can go either way.

  3. Turn the noise down

    In data.py, change rng.normal(0, 6, days) to rng.normal(0, 2, days). Predict what happens to the naive forecast, the seasonal naive forecast and the noise floor, then run python run.py. In my check of the baselines, the linear model and the noise floor, naive missed by 11.4, seasonal naive by 2.3, the 4 week mean by 2.1, the linear model by 1.9 and the noise floor by 1.6. I did not check gradient boosting.

    A hint

    Less noise helps the methods that use the same weekday more than the one that uses yesterday.

  4. Try a shorter test window

    In run.py, change both 56 values (in test and in train) to 28. Predict whether the order of the baselines changes, then run python run.py. In my check of the baselines, the linear model and the noise floor, naive missed by 12.6, seasonal naive by 6.9, the 4 week mean by 6.3, the linear model by 6.3 and the noise floor by 5.0. The video says one test window may not settle small gaps.

    A hint

    The video warns that gaps of a few tenths could change order on another window.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up, clean weekly data; one-day-ahead forecasts over one eight-week window; gradient boosting untuned; no classical models.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos