#67 LLM engineering and evals 6.8 min
Distillation: Teach a Small Model From a Big One (Python)
A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.
Run the four students
Copy the code folder first. Predict which of the four students will do best on the 200 test messages and which will do worst. Then run python run.py. It uses only the standard library and takes a couple of seconds. You should see 149 for 12 hand labels, 188 for the teacher's soft scores, 181 for the teacher's scores rounded to 0 or 1, and 200 for the true labels. Write one line on what surprised you.
A hint
The teacher itself is right on 185 of 200 in the video's run.
Give the teacher fewer neighbours
In teacher.py change k=7 to k=1 in teacher_prob, so the teacher copies only its single closest message. Predict whether the teacher gets better or worse, then run python run.py. You should see the teacher right on 183 of 200, the student from soft scores right on 190, and the student from 12 hand labels unchanged at 149.
A hint
The hand label student never uses the teacher, so its score should not change.
Give the student less to learn from
In run.py change UNLABELLED = make_set(300, seed=3) to make_set(30, seed=3). Predict what happens to the students trained on the teacher's labels, then run python run.py. You should see the soft score student fall to 169 of 200 and the rounded student to 157 of 200, so fewer unlabelled messages gave a worse student.
A hint
Only the students that use the unlabelled messages should change.
Read the teacher's mistakes
Run python misses.py and read the three messages the teacher gets wrong. Each is urgent but also carries a long calm phrase. In your own words, explain why a teacher that counts how many of its 7 closest messages are urgent might say calm for these. Then add the line print(len(wrong)) at the end of misses.py and run it again. You should see 15, which matches the teacher being right on 185 of 200.
A hint
The teacher was right on 185 of 200, so it gets 15 wrong.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
No language model is run: the teacher (nearest neighbours) and the student (logistic regression) are tiny models I wrote, on made-up messages with a simple rule; the student beating its teacher is a property of this toy, and a student trained on the teacher's own 400 true labels scored 200 of 200, so distillation only helped because those labels were set aside; 91 of the 200 test messages also appear verbatim in the teacher's memory or the labelled sets.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #182 Linear Regression From Scratch, Then scikit-learn Machine learning and data science
- #183 Precision, Recall, F1: Which Metric Matters? Machine learning and data science
- #184 Overfitting You Can See: Train vs Validation Error Machine learning and data science
- #185 Cross-Validation Done Right: The Leakage Mistake Machine learning and data science
