#57 LLM engineering and evals 5 min
LLM as Judge: I Swapped the Order and It Changed
I swapped the order of two answers and my judge changed its mind on half the questions in one test.
Read a case and its four answers
Copy the code folder first. Open cases.py and take the first question about returning a bike. Write down the source passage, the right short answer, the wrong short answer, and say which single figure was changed in the wrong answers. Then compare the wrong answers in the other cases with the right ones. Most change a figure, but the delivery case says 'Nothing' and 'Delivery is free', and some long wrong answers add a clause that the source passage does not contain (for example 'parts are priced first' for the full service, and 'cannot be swapped for cash' for the loyalty points).
A hint
The video says the wrong answers change a figure, but look closely: not every wrong answer changes only a figure.
Predict the first slot count
Read compare.py. In your own words, explain what the variable called first counts, and what value it would reach if the judge always answered A. Write your answer as a number out of 12 for one arm.
A hint
In one order the right answer is listed first, and in the other it is listed second. Count which slot a judge that always says A would pick.
Make the judge stricter
In judge.py change the end of the prompt so it asks for the answer that matches the source most closely, keeping the one-letter reply. Predict whether the number right in both orders will go up or down. If Ollama and llama3.2 are installed, run python compare.py and compare with the video, where the short-right arm was 3 of 6 right in both orders. Change the prompt back afterwards.
A hint
Change only the wording of the question. Keep temperature 0 and seed 1, so a repeat run should match.
Add a seventh question
Add a seventh tuple to CASES in cases.py with a short source passage you write, a question, a right and wrong answer in short and long form, where the wrong answer changes only a figure. Before running, predict whether the judge will get it right in both orders. If you can run Ollama, run python compare.py and see that the totals now use 7 questions.
A hint
Keep the order of the six fields the same as the comment at the top of CASES.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Six questions that I wrote, one small judge model (llama3.2), one run, so a difference of one is small. The wrong answers only change a figure. No larger judge, no other prompt and no model judging its own answers was tested; I do not claim long answers are always favoured. Any reply that does not start with B is counted as A; the raw replies were not printed.
Up next in LLM engineering and evals
More in LLM engineering and evals
Related videos
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
- #64 Chain of Thought: When Step by Step Helps, and Its Cost LLM engineering and evals
- #68 Synthetic Test Cases: 27 Asked, 17 Passed Checks LLM engineering and evals
