learning-it by example

Back to the video

A Neural Network From Scratch in NumPy, Gradient Checked

On 200 made-up points, a hidden layer took training accuracy from 0.59 to 0.99, and a gradient check caught a bug.

Video #196 ยท Machine learning and data science

Steps in the video

  1. 0:00 A hidden layer: accuracy 0.59 to 0.99, and a gradient check that caught a bug
  2. 0:30 The idea
  3. 0:59 Setup
  4. 1:21 The made-up points
  5. 1:37 Forward
  6. 2:00 The loss
  7. 2:10 Backward
  8. 2:33 Finite differences
  9. 2:53 Relative error
  10. 3:42 Training
  11. 4:51 What this does not show
  12. 5:20 Mistakes to avoid
  13. 5:44 My take and next steps
  14. 6:06 Recap

Files

  • check.py (29 lines)
  • data.py (9 lines)
  • net.py (31 lines)
  • run.py (28 lines)
  • train.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4

What this does not show

A toy network on 200 made-up 2D points; accuracy is on the training points only, with one seed; the gradient check ran once, at a random start (seed 2); the bug was put in on purpose; NumPy only, no deep learning framework.

Sources

Learning IT by Example.