learning-it by example

#67 LLM engineering and evals 6.8 min

Distillation: Teach a Small Model From a Big One (Python)

A small model trained on 12 hand labels got 149 of 200 right; trained on a teacher's labels, 188.

1. What is the recipe for distilling a big model into a small one, as the video describes it?
2. In the made-up urgent message test, how many of 200 test messages were right for the student trained on 12 hand labels, and for the student trained on the teacher's soft scores?
3. What did the video find when the teacher's soft scores were rounded to 0 or 1 before training the student?
4. The student beat its teacher, 188 against 185. Why can the video not claim that distillation makes a student better than its teacher?
5. Why does the video advise keeping a small set with true labels from people, even when a teacher does most of the labelling?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the four students

    Copy the code folder first. Predict which of the four students will do best on the 200 test messages and which will do worst. Then run python run.py. It uses only the standard library and takes a couple of seconds. You should see 149 for 12 hand labels, 188 for the teacher's soft scores, 181 for the teacher's scores rounded to 0 or 1, and 200 for the true labels. Write one line on what surprised you.

    A hint

    The teacher itself is right on 185 of 200 in the video's run.

  2. Give the teacher fewer neighbours

    In teacher.py change k=7 to k=1 in teacher_prob, so the teacher copies only its single closest message. Predict whether the teacher gets better or worse, then run python run.py. You should see the teacher right on 183 of 200, the student from soft scores right on 190, and the student from 12 hand labels unchanged at 149.

    A hint

    The hand label student never uses the teacher, so its score should not change.

  3. Give the student less to learn from

    In run.py change UNLABELLED = make_set(300, seed=3) to make_set(30, seed=3). Predict what happens to the students trained on the teacher's labels, then run python run.py. You should see the soft score student fall to 169 of 200 and the rounded student to 157 of 200, so fewer unlabelled messages gave a worse student.

    A hint

    Only the students that use the unlabelled messages should change.

  4. Read the teacher's mistakes

    Run python misses.py and read the three messages the teacher gets wrong. Each is urgent but also carries a long calm phrase. In your own words, explain why a teacher that counts how many of its 7 closest messages are urgent might say calm for these. Then add the line print(len(wrong)) at the end of misses.py and run it again. You should see 15, which matches the teacher being right on 185 of 200.

    A hint

    The teacher was right on 185 of 200, so it gets 15 wrong.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No language model is run: the teacher (nearest neighbours) and the student (logistic regression) are tiny models I wrote, on made-up messages with a simple rule; the student beating its teacher is a property of this toy, and a student trained on the teacher's own 400 true labels scored 200 of 200, so distillation only helped because those labels were set aside; 91 of the 200 test messages also appear verbatim in the teacher's memory or the labelled sets.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos