learning-it by example

Back to the video

Distillation: Teach a Small Model From a Big One (Python)

A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.

Video #67 ยท LLM engineering and evals

Steps in the video

  1. 0:00 12 hand labels: 149 of 200 right. The teacher's labels: 188.
  2. 0:32 The idea
  3. 1:11 Setup
  4. 1:38 The made-up messages
  5. 1:52 The teacher
  6. 2:17 The student
  7. 2:42 Four students
  8. 4:40 What this does not show
  9. 5:30 Mistakes to avoid
  10. 5:57 My take and next steps
  11. 6:19 Recap

Files

  • data.py (24 lines)
  • misses.py (10 lines)
  • run.py (35 lines)
  • student.py (19 lines)
  • teacher.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

No language model is run: the teacher (nearest neighbours) and the student (logistic regression) are tiny models I wrote, on made-up messages with a simple rule; the student beating its teacher is a property of this toy, and a student trained on the teacher's own 400 true labels scored 200 of 200, so distillation only helped because those labels were set aside; 91 of the 200 test messages also appear verbatim in the teacher's memory or the labelled sets.

Sources

Learning IT by Example.