Distillation: Teach a Small Model From a Big One (Python)
A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.
Video #67 ยท LLM engineering and evals
Steps in the video
Files
- data.py (24 lines)
- misses.py (10 lines)
- run.py (35 lines)
- student.py (19 lines)
- teacher.py (16 lines)
Tested with
Python 3.12.15
What this does not show
No language model is run: the teacher (nearest neighbours) and the student (logistic regression) are tiny models I wrote, on made-up messages with a simple rule; the student beating its teacher is a property of this toy, and a student trained on the teacher's own 400 true labels scored 200 of 200, so distillation only helped because those labels were set aside; 91 of the 200 test messages also appear verbatim in the teacher's memory or the labelled sets.
Sources
Learning IT by Example.