learning-it by example

#57 LLM engineering and evals 5 min

LLM as Judge: I Swapped the Order and It Changed

I swapped the order of two answers and my judge changed its mind on half the questions in one test.

1. How do you test whether a judge model can be trusted to pick the better of two answers?
2. In the video's LLM-as-judge test (llama3.2 judging six questions the author wrote, each pair tried in both orders), the right answer was short and the wrong answer long, and the judge picked the right answer 9 times out of 12. How many of the 6 questions were right in both orders?
3. In the video's LLM-as-judge test (llama3.2, six questions the author wrote, each pair tried in both orders), how did the judge do when the right answer was long and the wrong answer short?
4. What limits did the video state about its llama3.2 judge test?
5. In the video's LLM-as-judge test, how is the judge model asked to answer?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read a case and its four answers

    Copy the code folder first. Open cases.py and take the first question about returning a bike. Write down the source passage, the right short answer, the wrong short answer, and say which single figure was changed in the wrong answers. Then compare the wrong answers in the other cases with the right ones. Most change a figure, but the delivery case says 'Nothing' and 'Delivery is free', and some long wrong answers add a clause that the source passage does not contain (for example 'parts are priced first' for the full service, and 'cannot be swapped for cash' for the loyalty points).

    A hint

    The video says the wrong answers change a figure, but look closely: not every wrong answer changes only a figure.

  2. Predict the first slot count

    Read compare.py. In your own words, explain what the variable called first counts, and what value it would reach if the judge always answered A. Write your answer as a number out of 12 for one arm.

    A hint

    In one order the right answer is listed first, and in the other it is listed second. Count which slot a judge that always says A would pick.

  3. Make the judge stricter

    In judge.py change the end of the prompt so it asks for the answer that matches the source most closely, keeping the one-letter reply. Predict whether the number right in both orders will go up or down. If Ollama and llama3.2 are installed, run python compare.py and compare with the video, where the short-right arm was 3 of 6 right in both orders. Change the prompt back afterwards.

    A hint

    Change only the wording of the question. Keep temperature 0 and seed 1, so a repeat run should match.

  4. Add a seventh question

    Add a seventh tuple to CASES in cases.py with a short source passage you write, a question, a right and wrong answer in short and long form, where the wrong answer changes only a figure. Before running, predict whether the judge will get it right in both orders. If you can run Ollama, run python compare.py and see that the totals now use 7 questions.

    A hint

    Keep the order of the six fields the same as the comment at the top of CASES.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six questions that I wrote, one small judge model (llama3.2), one run, so a difference of one is small. The wrong answers only change a figure. No larger judge, no other prompt and no model judging its own answers was tested; I do not claim long answers are always favoured. Any reply that does not start with B is counted as A; the raw replies were not printed.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos