learning-it by example

#95 AI security and governance 5 min

Never Trust Model Output: Validate It in Python

All ten of my model replies passed every check, and three were still wrong.

1. What does OWASP guidance on improper output handling say to do with model output?
2. llama3.2 triaged ten made-up support tickets in free text, and a cleanup kept only the text between the first and last brace. What did the JSON and shape checks find?
3. In the llama3.2 ticket-triage test, what happened with the check that the refund must not be above the order total?
4. In the ticket-triage demo of the video 'Never trust model output: validate it', how is the pydantic Triage model used?
5. What does the video say to do before passing model output to a database, a shell or a web page?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict what check returns

    Copy the code folder first. In a Python session in that folder run: from check import check. Predict what check returns for each of these replies with an order total of 20: first check('{"category": "billing", "priority": 5, "refund_gbp": 10}', 20), then check('{"category": "billing", "priority": 2, "refund_gbp": 30}', 20), then check('{"category": "billing", "priority": 2, "refund_gbp": 10}', 20). Write your predictions down, then run them.

    A hint

    The three checks run in order and the function returns the first one that fails, or ok. Priority must be from 1 to 3.

  2. Try the brace cleanup by hand

    In a Python session define the cleanup from compare.py: def cleanup(text): return text[text.find('{'):text.rfind('}') + 1]. Predict what it returns for 'Sure! {"a": 1} Hope that helps.', for 'no braces here', and for '{"a": 1} and {"b": 2}'. Then run them. You should get the object alone, an empty string, and both objects with the text between them.

    A hint

    The video calls cleaning text with a pattern a mistake to treat as validation. Look at the third answer to see why.

  3. Tighten the refund rule

    In check.py change the comparison triage.refund_gbp > total to triage.refund_gbp >= total. Predict what check returns for a reply with priority 2 and a refund of 15 when the order total is 15, before and after the change (start a fresh Python session after editing check.py, because the open one still has the old function). Write your prediction down, then run it. Decide which version matches the rule in the video, which says the refund must not be above the order total. Then change it back to >, because the next task runs compare.py.

    A hint

    Equal is not above, so think about which version lets an equal refund through.

  4. Change a ticket label

    This needs Ollama with llama3.2, so treat it as a prediction. In tickets.py change the right category of the cracked blender ticket from technical to billing. Predict which printed numbers in compare.py can change and which cannot. Write your prediction down, then run it. The video calls this ticket a judgement call.

    A hint

    The prompt never mentions the right category, so the model's replies are the same. Only the count of right categories uses it.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Ten made-up tickets, one seed, one small local model (llama3.2). Right means only that the category matches the one the author chose; priority and refund are not scored. The refund rule never fired in this run, so it is not shown working on a bad reply. Validation only checks what you thought to check.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos