learning-it by example

#71 LLM engineering and evals 6.2 min

Model Routing: Send Easy Questions to a Small Model

Routing cost 19.1 made-up cents instead of 40, and got one more of 20 questions wrong.

  1. Halve the large model's price

    Copy the code folder first. Run python run.py compare and note the three lines. Then in models.py change the large price in PRICE from 2.0 to 1.0. Predict the three new cost figures before you run it again. The all large line should read 20.0 cents with 19 of 20 right, and the routed line should read 10.1 cents with 18 of 20 right. Which numbers did not change, and why?

    A hint

    The skill of each pretend model decides which answers are right, and the price only changes the cost.

  2. Give the small model more skill

    Restore the price, then in models.py change the small model's skill in SKILL from 2 to 3. Predict which of the three lines in python run.py compare change before you run it. You should see all small go from 11 to 14 of 20 right, with the cost unchanged at 2.0 cents, while the routed line stays at 19.1 cents and 18 of 20 right. Why does the router not take advantage of the better small model?

    A hint

    Look at router.py. Does the router ever look at the models' skill?

  3. Remove a signal word

    Restore models.py. In router.py delete the word why from HARD_WORDS. Predict whether the routed line at a threshold of 2 gets cheaper or more expensive, and whether it gets more or fewer answers right, then run python run.py compare and python run.py misses. You should see the routed line become cheaper, at 15.3 cents, and drop to 16 of 20 right, with two more wrong lines in the misses list. Write one sentence on why a cost figure on its own would hide this.

    A hint

    Two of the questions in questions.py start with Why. Check the guess they now get.

  4. Find the threshold that fixes the polite question

    Restore router.py. The long polite question about Japan has a guess of 2.2. In run.py change the thresholds in the sweep loop to (2.2, 2.3, 4) and run python run.py sweep. Predict which threshold stops that question going to the large model. You should see 19.1 cents at 2.2, 17.2 cents at 2.3 and 5.8 cents with only 13 of 20 right at 4. Then explain why picking 2.3 because of one known question would be choosing a threshold on the questions you report.

    A hint

    The router sends a question to the large model when its guess is at or above the threshold.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No model is called: questions, difficulties, prices and model skill are made up, so right and wrong are true by construction, and the threshold and signal words were tuned on the same twenty questions.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos