learning-it by example

Back to the video

Detect LLM Hallucinations: 5 Checks Scored on 16 Answers

One check caught 1 of 8 made up answers. A strict model judge flagged 6 of 8 good ones.

Video #69 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Five checks
  2. 0:31 The idea
  3. 1:07 Setup
  4. 1:34 The sixteen answers
  5. 2:04 Two cheap checks
  6. 2:19 Embeddings
  7. 2:35 Two judges
  8. 3:20 The judges
  9. 3:46 What the code checks missed
  10. 4:11 What this does not show
  11. 4:44 Mistakes to avoid
  12. 5:06 My take and next steps
  13. 5:37 Recap

Files

  • answers.py (31 lines)
  • checks.py (74 lines)
  • handbook.md (28 lines)
  • score.py (24 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Sixteen one-sentence answers that I wrote about one made-up handbook, one run, and one small local model (llama3.2) as the judge with two prompts; the 0.8 and 0.75 limits are my choice, and no claim splitting or larger judge was tested.

Sources

Learning IT by Example.