#296 LLM engineering and evals 6.4 min
Attention in 40 Lines of NumPy: The Core of Transformers
With hand-set weights, the word tired gave 0.79 of its attention to cat, in about 40 lines of NumPy.
Read the weight table
Copy the code folder first. This needs NumPy, which the video already requires. Run python run.py. Rows look and columns are looked at, so find the row for tired and say which word it looks at most. You should see cat with 0.79. Then check that every row adds up to about 1 (the table is rounded to two decimals, so a row can show 0.99 or 1.01).
A hint
Each row is one word's weights over all four words.
Take away the square root
In attention.py remove the division by np.sqrt(d_k) in the scores line, then run python run.py. Predict whether the weight tired gives to cat will go up or down. In a test run it went from 0.79 up to 0.91. Then run python run.py scale and read how the video shows the effect with 64 numbers per vector. Put the division by np.sqrt(d_k) back afterwards, because the next exercises quote results for the original code.
A hint
In this toy each query and key has only 2 numbers, so the effect is small. The scale mode uses 64.
Make tired ask harder
In words.py change the number 2 in the last row of W_q to 4, then run python run.py. Predict how the weight for tired looking at cat changes. On a test run it rose to 0.98. Then change it back.
A hint
A larger query gives a bigger score for the word that matches, and softmax makes the biggest score count more.
Scale the values
In words.py change W_v = np.eye(4) to W_v = np.eye(4) * 2 and run python run.py. Predict which numbers change: the weights, or the new vector for tired. On a test run the weights stayed the same and the new vector doubled in size to [0.12 1.61 0.15 0.52].
A hint
The new vector is the weights times the values.
Run the causal mask yourself
Run python run.py causal and check the claim that everything above the diagonal is zero. Then predict the first row of the table, and explain why the row for tired is the same as in the plain run.
A hint
The first word can only look at itself, and tired is the last word, so no later word is hidden from it.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
The words, numbers and weights are made up and set by hand; no model is trained or run, and it is one head only. The random-weights seed was picked from seeds 1 to 7 for its flat row; seed 2 is as flat, the other five put 0.37 to 0.51 on one word.
Up next in LLM engineering and evals
More in LLM engineering and evals
Related videos
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #184 Overfitting You Can See: Train vs Validation Error Machine learning and data science
- #194 Anomaly Detection Basics: z-Score, IQR, Isolation Forest Machine learning and data science
- #183 Precision, Recall, F1: Which Metric Matters? Machine learning and data science
