learning-it by example

#196 Machine learning and data science 6.5 min

A Neural Network From Scratch in NumPy, Gradient Checked

On 200 made-up points, a hidden layer took training accuracy from 0.59 to 0.99, and a gradient check caught a bug.

1. Why did the network with no hidden layer stay at an accuracy of 0.59 on the circle data?
2. On the gradient check of 33 parameters, what happened with the bug that skips the tanh slope?
3. What is the slope of tanh that the backward pass multiplies by?
4. With 8 tanh hidden units, what were the loss and the accuracy on the training points after 2000 steps?
5. Why does the 0.99 accuracy of the NumPy hidden-layer network on the circle points say little about how it would do on new points?
6. How does the gradient check estimate the gradient of one weight?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fewer hidden units

    Copy the code folder first. In run.py change for hidden in (8, 0) to for hidden in (2, 0) and run python run.py train. You should see that with 2 hidden units the loss ends near 0.443 and the accuracy near 0.79 at step 2000, well below 0.99 with 8 units but better than 0.59 with none.

    A hint

    Only the number of hidden units changes.

  2. A smaller learning rate

    In train.py change lr=0.5 to lr=0.05 and run python run.py train. Predict first whether the 8-unit network will reach the same loss by step 2000. You should see it reach loss 0.265 and accuracy 0.97 at step 2000, still improving, and only 0.64 accuracy at step 500.

    A hint

    The learning rate is the size of each step downhill.

  3. Make the check less careful

    In check.py change h=1e-5 to h=1e-2 in the numeric_grads function definition. The first printed line will still say h = 1e-5, because that text is fixed. Predict whether the correct backward will still pass the 1e-7 line, then run python run.py check. You should see the correct backward fail the line, with a worst value printed as 0.00, while the bug still shows worst 1.78.

    A hint

    The printed value is rounded to two decimals, so 0.00 can still be above 1e-7.

  4. Train with the bug switched on

    In train.py change grads = backward(p, X, y) to grads = backward(p, X, y, bug=True) and run python run.py train. Predict whether the 8-unit network still learns the circle. You should see it stall near loss 0.66 and accuracy 0.59 or 0.60, with no error message.

    A hint

    This is the bug that skips the tanh slope, the one the gradient check caught.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A toy network on 200 made-up 2D points; accuracy is on the training points only, with one seed; the gradient check ran once, at a random start (seed 2); the bug was put in on purpose; NumPy only, no deep learning framework.

Up next in Machine learning and data science

More in Machine learning and data science

All Machine learning and data science videos

Related videos