#196 Machine learning and data science 6.5 min
A Neural Network From Scratch in NumPy, Gradient Checked
On 200 made-up points, a hidden layer took training accuracy from 0.59 to 0.99, and a gradient check caught a bug.
Fewer hidden units
Copy the code folder first. In run.py change for hidden in (8, 0) to for hidden in (2, 0) and run python run.py train. You should see that with 2 hidden units the loss ends near 0.443 and the accuracy near 0.79 at step 2000, well below 0.99 with 8 units but better than 0.59 with none.
A hint
Only the number of hidden units changes.
A smaller learning rate
In train.py change lr=0.5 to lr=0.05 and run python run.py train. Predict first whether the 8-unit network will reach the same loss by step 2000. You should see it reach loss 0.265 and accuracy 0.97 at step 2000, still improving, and only 0.64 accuracy at step 500.
A hint
The learning rate is the size of each step downhill.
Make the check less careful
In check.py change h=1e-5 to h=1e-2 in the numeric_grads function definition. The first printed line will still say h = 1e-5, because that text is fixed. Predict whether the correct backward will still pass the 1e-7 line, then run python run.py check. You should see the correct backward fail the line, with a worst value printed as 0.00, while the bug still shows worst 1.78.
A hint
The printed value is rounded to two decimals, so 0.00 can still be above 1e-7.
Train with the bug switched on
In train.py change grads = backward(p, X, y) to grads = backward(p, X, y, bug=True) and run python run.py train. Predict whether the 8-unit network still learns the circle. You should see it stall near loss 0.66 and accuracy 0.59 or 0.60, with no error message.
A hint
This is the bug that skips the tanh slope, the one the gradient check caught.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A toy network on 200 made-up 2D points; accuracy is on the training points only, with one seed; the gradient check ran once, at a random start (seed 2); the bug was put in on purpose; NumPy only, no deep learning framework.
Up next in Machine learning and data science
More in Machine learning and data science
- #182 Linear Regression From Scratch, Then scikit-learn
- #183 Precision, Recall, F1: Which Metric Matters?
- #184 Overfitting You Can See: Train vs Validation Error
- #185 Cross-Validation Done Right: The Leakage Mistake
- #186 Data Leakage: The Bug That Fakes Accuracy, Reproduced
- #187 Feature Engineering Beat 500 Trees, on Made-Up Data
Related videos
- #195 Recommender From Scratch: Watch 24 Films Find Their Genres Machine learning and data science
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search
- #03 pgvector vs Chroma vs Qdrant: Same Data, Real Results RAG and vector search
