Prompt Versioning: Compare Changes Case by Case
I changed a prompt to make it better and the score went down. A score alone could not tell me which cases broke.
Video #56 ยท LLM engineering and evals
Steps in the video
Files
- cases.py (17 lines)
- compare.py (29 lines)
- edit_prompt.py (8 lines)
- prompts/v1.txt (1 lines)
- prompts/v2.txt (1 lines)
- prompts/v3.txt (1 lines)
- score.py (38 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Fourteen cases that I wrote and labelled myself (one label is arguable), so one case is 7 percent. One small model (llama3.2), with no model snapshot pinned, and one run. The prompts were not tuned.
Sources
Learning IT by Example.