learning-it by example

Back to the video

Prompt Versioning: Compare Changes Case by Case

I changed a prompt to make it better and the score went down. A score alone could not tell me which cases broke.

Video #56 ยท LLM engineering and evals

Steps in the video

  1. 0:00 The change that lost points
  2. 0:20 The idea
  3. 0:49 Setup
  4. 1:04 The cases
  5. 1:17 Three versions
  6. 1:38 Scoring a version
  7. 2:25 The comparison
  8. 3:04 A stale score
  9. 3:41 What this does not show
  10. 4:07 Mistakes to avoid
  11. 4:30 My take and next steps
  12. 4:54 Recap

Files

  • cases.py (17 lines)
  • compare.py (29 lines)
  • edit_prompt.py (8 lines)
  • prompts/v1.txt (1 lines)
  • prompts/v2.txt (1 lines)
  • prompts/v3.txt (1 lines)
  • score.py (38 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Fourteen cases that I wrote and labelled myself (one label is arguable), so one case is 7 percent. One small model (llama3.2), with no model snapshot pinned, and one run. The prompts were not tuned.

Sources

Learning IT by Example.