learning-it by example

Back to the video

Model Routing: Send Easy Questions to a Small Model

Routing cost 19.1 made-up cents instead of 40, and got one more of 20 questions wrong.

Video #71 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Routing cost 19.1 cents, not 40. It got one more question wrong.
  2. 0:34 The idea
  3. 1:12 Setup
  4. 1:40 The questions
  5. 2:06 The pretend models
  6. 2:33 The router
  7. 3:04 Counting the cost
  8. 4:34 What this does not show
  9. 5:00 Mistakes to avoid
  10. 5:25 My take and next steps
  11. 5:51 Recap

Files

  • models.py (7 lines)
  • questions.py (24 lines)
  • router.py (13 lines)
  • run.py (43 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

No model is called: questions, difficulties, prices and model skill are made up, so right and wrong are true by construction, and the threshold and signal words were tuned on the same twenty questions.

Sources

Learning IT by Example.