A Neural Network From Scratch in NumPy, Gradient Checked
On 200 made-up points, a hidden layer took training accuracy from 0.59 to 0.99, and a gradient check caught a bug.
Video #196 ยท Machine learning and data science
Steps in the video
- 0:00 A hidden layer: accuracy 0.59 to 0.99, and a gradient check that caught a bug
- 0:30 The idea
- 0:59 Setup
- 1:21 The made-up points
- 1:37 Forward
- 2:00 The loss
- 2:10 Backward
- 2:33 Finite differences
- 2:53 Relative error
- 3:42 Training
- 4:51 What this does not show
- 5:20 Mistakes to avoid
- 5:44 My take and next steps
- 6:06 Recap
Files
- check.py (29 lines)
- data.py (9 lines)
- net.py (31 lines)
- run.py (28 lines)
- train.py (16 lines)
Tested with
Python 3.12.15 | numpy==2.5.4
What this does not show
A toy network on 200 made-up 2D points; accuracy is on the training points only, with one seed; the gradient check ran once, at a random start (seed 2); the bug was put in on purpose; NumPy only, no deep learning framework.
Sources
Learning IT by Example.