learning-it by example

#56 LLM engineering and evals 5.3 min

Prompt Versioning: Compare Changes Case by Case

I changed a prompt to make it better and the score went down. A score alone could not tell me which cases broke.

1. In prompt versioning, what do you record with each scored run so the score is not pinned on the wrong prompt?
2. On the fourteen email triage cases, which prompt versions tied on score, and what were the totals?
3. Going from prompt version 1 to version 2, the score fell by one point. What did the case by case comparison show?
4. Why does the comparison script refuse to compare a prompt version that was edited after it was scored?
5. What is a mistake to avoid when changing a prompt with a rule such as 'if the email mentions money, always choose billing'?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the golden cases

    Copy the code folder first. Open cases.py and read the fourteen emails and the category chosen for each. Pick the one label you find most arguable and write down why. Then count how many cases are billing, bug, feature and other.

    A hint

    The video said one of its labels is arguable, and that no prompt version got the blank invoice PDF case right.

  2. Predict a fingerprint change

    Open prompts/v1.txt. Without running anything, predict whether the fingerprint made by fingerprint() would change if you added one full stop at the end of the text, and whether it would change if you added a blank line after the text. Then read score.py to check how the text is prepared before it is hashed.

    A hint

    Look at the .strip() call in score.py before the text is hashed.

  3. Write your own version

    Create prompts/v4.txt by copying v1.txt and adding one sentence of your own, such as a rule for emails that ask a question about the account. Predict which of the fourteen cases it could fix and which it could break. If Ollama and the llama3.2 model are installed, run python score.py v1, then python score.py v4 and then python compare.py v1 v4 and check your prediction.

    A hint

    Do not edit v1.txt itself, or the comparison will refuse because the fingerprint no longer matches.

  4. Trigger the stale score refusal

    If you have scored v1 and v2, run python edit_prompt.py to make an edited copy, then python compare.py v1 v2 edited. Predict what the refusal message will say, and which two fingerprints it will show. Then explain in one sentence what you would have to do to compare the edited v2 fairly.

    A hint

    The edit only exists in the edited folder, not in the real prompts folder.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Fourteen cases that I wrote and labelled myself (one label is arguable), so one case is 7 percent. One small model (llama3.2), with no model snapshot pinned, and one run. The prompts were not tuned.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos