learning-it by example

Back to the video

Attention in 40 Lines of NumPy: The Core of Transformers

With hand-set weights, the word tired gave 0.79 of its attention to cat, in about 40 lines of NumPy.

Video #296 ยท LLM engineering and evals

Steps in the video

  1. 0:00 The word tired put 0.79 of its attention on cat
  2. 0:32 The idea
  3. 1:03 Setup
  4. 1:22 Four made-up words
  5. 1:48 Softmax
  6. 2:05 One head of attention
  7. 2:33 Run it
  8. 4:05 Why divide by the square root
  9. 5:04 What this does not show
  10. 5:29 Mistakes to avoid
  11. 5:50 My take and next steps
  12. 6:07 Recap

Files

  • attention.py (22 lines)
  • run.py (39 lines)
  • words.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | numpy==2.5.4

What this does not show

The words, numbers and weights are made up and set by hand; no model is trained or run, and it is one head only. The random-weights seed was picked from seeds 1 to 7 for its flat row; seed 2 is as flat, the other five put 0.37 to 0.51 on one word.

Sources

Learning IT by Example.