learning-it by example

Back to the video

LLM as Judge: I Swapped the Order and It Changed

I swapped the order of two answers and my judge changed its mind on half the questions in one test.

Video #57 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Can you trust a judge model?
  2. 0:27 The idea
  3. 1:06 Setup
  4. 1:22 Six cases
  5. 1:42 The judge
  6. 1:59 Both orders
  7. 3:00 What this does not show
  8. 3:28 Mistakes to avoid
  9. 3:53 My take and next steps
  10. 4:32 Recap

Files

  • cases.py (47 lines)
  • compare.py (22 lines)
  • judge.py (12 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Six questions that I wrote, one small judge model (llama3.2), one run, so a difference of one is small. The wrong answers only change a figure. No larger judge, no other prompt and no model judging its own answers was tested; I do not claim long answers are always favoured. Any reply that does not start with B is counted as A; the raw replies were not printed.

Sources

Learning IT by Example.