learning-it by example

Back to the video

Never Trust Model Output: Validate It in Python

All ten of my model replies passed every check, and three were still wrong.

Video #95 ยท AI security and governance

Steps in the video

  1. 0:00 All ten replies passed, and three were still wrong
  2. 0:25 The idea
  3. 0:58 Setup
  4. 1:17 The shape
  5. 1:38 Three ways to ask
  6. 2:00 Three checks
  7. 2:21 Ten tickets
  8. 3:15 What this does not show
  9. 3:39 Mistakes to avoid
  10. 4:05 My take and next steps
  11. 4:30 Recap

Files

  • ask.py (15 lines)
  • check.py (20 lines)
  • compare.py (30 lines)
  • schema.py (10 lines)
  • tickets.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Ten made-up tickets, one seed, one small local model (llama3.2). Right means only that the category matches the one the author chose; priority and refund are not scored. The refund rule never fired in this run, so it is not shown working on a bad reply. Validation only checks what you thought to check.

Sources

Learning IT by Example.