#95 AI security and governance 5 min
Never Trust Model Output: Validate It in Python
All ten of my model replies passed every check, and three were still wrong.
Predict what check returns
Copy the code folder first. In a Python session in that folder run: from check import check. Predict what check returns for each of these replies with an order total of 20: first check('{"category": "billing", "priority": 5, "refund_gbp": 10}', 20), then check('{"category": "billing", "priority": 2, "refund_gbp": 30}', 20), then check('{"category": "billing", "priority": 2, "refund_gbp": 10}', 20). Write your predictions down, then run them.
A hint
The three checks run in order and the function returns the first one that fails, or ok. Priority must be from 1 to 3.
Try the brace cleanup by hand
In a Python session define the cleanup from compare.py: def cleanup(text): return text[text.find('{'):text.rfind('}') + 1]. Predict what it returns for 'Sure! {"a": 1} Hope that helps.', for 'no braces here', and for '{"a": 1} and {"b": 2}'. Then run them. You should get the object alone, an empty string, and both objects with the text between them.
A hint
The video calls cleaning text with a pattern a mistake to treat as validation. Look at the third answer to see why.
Tighten the refund rule
In check.py change the comparison triage.refund_gbp > total to triage.refund_gbp >= total. Predict what check returns for a reply with priority 2 and a refund of 15 when the order total is 15, before and after the change (start a fresh Python session after editing check.py, because the open one still has the old function). Write your prediction down, then run it. Decide which version matches the rule in the video, which says the refund must not be above the order total. Then change it back to >, because the next task runs compare.py.
A hint
Equal is not above, so think about which version lets an equal refund through.
Change a ticket label
This needs Ollama with llama3.2, so treat it as a prediction. In tickets.py change the right category of the cracked blender ticket from technical to billing. Predict which printed numbers in compare.py can change and which cannot. Write your prediction down, then run it. The video calls this ticket a judgement call.
A hint
The prompt never mentions the right category, so the model's replies are the same. Only the count of right categories uses it.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Ten made-up tickets, one seed, one small local model (llama3.2). Right means only that the category matches the one the author chose; priority and refund are not scored. The refund rule never fired in this run, so it is not shown working on a bad reply. Validation only checks what you thought to check.
Up next in AI security and governance
More in AI security and governance
- #18 Prompt Injection Live: Attacking My Own LLM App
- #19 Redact Personal Data Before Your LLM Sees It (Python)
- #88 Indirect Prompt Injection: A Note That Gave Orders
- #89 Jailbreak-Test Your Own App: Eight Wrappers, Counted
- #90 Stop Leaking API Keys: Build a Secret Scanner in Python
- #91 Rate Limiting an LLM API: Token Bucket in FastAPI
Related videos
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama) LLM engineering and evals
- #26 Plan, Act, Reflect: Does It Help a Small Model? AI agents and MCP
- #88 Indirect Prompt Injection: A Note That Gave Orders AI security and governance
- #94 Can Your Model Leak Its System Prompt? I Tried 8 Ways AI security and governance
