LLM as Judge: I Swapped the Order and It Changed
I swapped the order of two answers and my judge changed its mind on half the questions in one test.
Video #57 ยท LLM engineering and evals
Steps in the video
Files
- cases.py (47 lines)
- compare.py (22 lines)
- judge.py (12 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Six questions that I wrote, one small judge model (llama3.2), one run, so a difference of one is small. The wrong answers only change a figure. No larger judge, no other prompt and no model judging its own answers was tested; I do not claim long answers are always favoured. Any reply that does not start with B is counted as A; the raw replies were not printed.
Sources
Learning IT by Example.