#71 LLM engineering and evals 6.2 min
Model Routing: Send Easy Questions to a Small Model
Routing cost 19.1 made-up cents instead of 40, and got one more of 20 questions wrong.
Halve the large model's price
Copy the code folder first. Run python run.py compare and note the three lines. Then in models.py change the large price in PRICE from 2.0 to 1.0. Predict the three new cost figures before you run it again. The all large line should read 20.0 cents with 19 of 20 right, and the routed line should read 10.1 cents with 18 of 20 right. Which numbers did not change, and why?
A hint
The skill of each pretend model decides which answers are right, and the price only changes the cost.
Give the small model more skill
Restore the price, then in models.py change the small model's skill in SKILL from 2 to 3. Predict which of the three lines in python run.py compare change before you run it. You should see all small go from 11 to 14 of 20 right, with the cost unchanged at 2.0 cents, while the routed line stays at 19.1 cents and 18 of 20 right. Why does the router not take advantage of the better small model?
A hint
Look at router.py. Does the router ever look at the models' skill?
Remove a signal word
Restore models.py. In router.py delete the word why from HARD_WORDS. Predict whether the routed line at a threshold of 2 gets cheaper or more expensive, and whether it gets more or fewer answers right, then run python run.py compare and python run.py misses. You should see the routed line become cheaper, at 15.3 cents, and drop to 16 of 20 right, with two more wrong lines in the misses list. Write one sentence on why a cost figure on its own would hide this.
A hint
Two of the questions in questions.py start with Why. Check the guess they now get.
Find the threshold that fixes the polite question
Restore router.py. The long polite question about Japan has a guess of 2.2. In run.py change the thresholds in the sweep loop to (2.2, 2.3, 4) and run python run.py sweep. Predict which threshold stops that question going to the large model. You should see 19.1 cents at 2.2, 17.2 cents at 2.3 and 5.8 cents with only 13 of 20 right at 4. Then explain why picking 2.3 because of one known question would be choosing a threshold on the questions you report.
A hint
The router sends a question to the large model when its guess is at or above the threshold.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
No model is called: questions, difficulties, prices and model skill are made up, so right and wrong are true by construction, and the threshold and signal words were tuned on the same twenty questions.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #244 Cost Control for AI Agents: A Budget for Each Agent Enterprise architecture and AI agents
- #12 Multi-Agent Patterns in Plain Python: Pipeline, Router, Loop AI agents and MCP
- #318 LLM Latency and Cost Budget: Which Lever Matters? Forward deployed engineer interviews
- #53 Semantic Cache for LLM Apps: Pick the Threshold (Python) RAG and vector search
