Attention in 40 Lines of NumPy: The Core of Transformers
With hand-set weights, the word tired gave 0.79 of its attention to cat, in about 40 lines of NumPy.
Video #296 ยท LLM engineering and evals
Steps in the video
Files
- attention.py (22 lines)
- run.py (39 lines)
- words.py (16 lines)
Tested with
Python 3.12.15 | numpy==2.5.4
What this does not show
The words, numbers and weights are made up and set by hand; no model is trained or run, and it is one head only. The random-weights seed was picked from seeds 1 to 7 for its flat row; seed 2 is as flat, the other five put 0.37 to 0.51 on one word.
Sources
Learning IT by Example.