Detect LLM Hallucinations: 5 Checks Scored on 16 Answers
One check caught 1 of 8 made up answers. A strict model judge flagged 6 of 8 good ones.
Video #69 ยท LLM engineering and evals
Steps in the video
Files
- answers.py (31 lines)
- checks.py (74 lines)
- handbook.md (28 lines)
- score.py (24 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Sixteen one-sentence answers that I wrote about one made-up handbook, one run, and one small local model (llama3.2) as the judge with two prompts; the 0.8 and 0.75 limits are my choice, and no claim splitting or larger judge was tested.
Sources
Learning IT by Example.