learning-it by example

#189 Machine learning and data science 6.3 min

Random Forest Explained in Code: Bagging by Hand

One deep tree scored 0.728 on test; a hundred random trees together scored 0.842.

1. What two kinds of randomness does a random forest add to single decision trees?
2. In the made-up circle data, one full decision tree got what accuracy on the training points and on the test points?
3. How does the hand-built forest turn the trees' answers into one prediction?
4. The hand-built random forest of 100 trees scored 0.842 on the made-up circle data and scikit-learn's forest of 100 trees scored 0.828. What should you conclude about that gap?
5. In the made-up circle data, what did the single full decision tree's training accuracy of 1.000 tell you?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run and compare the forests

    Copy the code folder first. This video needs numpy and scikit-learn, as listed in requirements.txt. Run python run.py. Compare the test accuracy of the one full tree with your forest of 1, 5, 25 and 100 trees. In the video's run, the single tree scored 0.728 on test and your hundred-tree forest scored 0.842.

    A hint

    The first line after the data line also shows the training accuracy of the single tree.

  2. Predict more trees

    In run.py, change the list [1, 5, 25, 100] to [1, 5, 25, 100, 400]. Predict whether the accuracy with 400 trees will be much higher than with 100, then run it and compare. The video quotes the docs: results stop getting much better beyond a critical number of trees.

    A hint

    Look at how much the score changed from 25 to 100 trees.

  3. Predict with no flipped labels

    In data.py, change flip=0.1 to flip=0.0, so no label is wrong on purpose. Predict what happens to the test accuracy of the single tree and of the forest of 100 trees, then run python run.py and compare. Say in one sentence why the flipped labels made things harder.

    A hint

    With flipped labels, even a perfect model would miss about one in ten.

  4. Change the forest seed

    In forest.py, the function fit_forest has seed=0. Change it to seed=1 and run python run.py again. Predict which lines change, then compare with the first run. What does this tell you about comparing two forests on one seed?

    A hint

    The seed decides which rows each tree draws.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Made-up data, one train and test split, one seed, and no tuning; the trees themselves come from scikit-learn.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos