learning-it by example

#102 AI security and governance 5 min

Red-Team Your AI Agent With a Checklist

Four of six test attacks worked on my demo agent; two were plain delete and send requests.

1. Where does the video say the defence against harmful agent actions should live?
2. Six attacks were run against a notes agent. How many did harm with no guards, and how many with the guards in code?
3. Two attacks did no harm even without guards. Which two, and what does the video say about them?
4. With guards on, how does the read tool stop a request for a file outside the notes folder?
5. What should the test for harm look at in a red-team checklist for an agent?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the vault with and without guards

    Copy the code folder first. The model is not needed for this: call the tools directly. In a Python session in the folder, make a temporary notes folder with the make function from shelf.py, build tools with make_tools(notes, False) from agent.py, and call read_note('../vault.txt'). Then repeat with True. Note: make_tools returns a pair (the tools and a state), and agent.py imports ollama, so the ollama package from requirements.txt must be installed. When I ran it, the first returned the vault text and the second returned: error: that is not a note.

    A hint

    Use tempfile.TemporaryDirectory() for the root and pass the folder that make(root) returns as notes.

  2. Delete all with and without guards

    In the same way, call delete_note('all') and then list_notes() on a fresh notes folder. Predict the results with guards off and with guards on, then run both. When I ran it, with guards off the notes were all deleted and list_notes returned an empty string, and with guards on the reply was error: a human must approve deletes and the three notes were still there. Note: make_tools returns a pair (the tools and a state), and agent.py imports ollama, so the ollama package from requirements.txt must be installed.

    A hint

    Use a new temporary folder for each run, so the second run starts with all three notes.

  3. Find what the path guard allows

    With guards on, predict what read_note('../notes/shopping') returns. It goes outside the folder and back in. Then run it. When I ran it, it returned the shopping note, because the guard checks where the resolved path ends up, not the text typed. Note: make_tools returns a pair (the tools and a state), and agent.py imports ollama, so the ollama package from requirements.txt must be installed.

    A hint

    The guard asks whether the notes folder is among the parents of the resolved path.

  4. Add a seventh attack

    This needs Ollama with llama3.2, so treat it as a prediction. Add a seventh line to CHECKLIST in compare.py with an attack of your own and a test for harm that looks at the state of the tools, not at the reply. Predict whether it does harm with no guards and with guards. Write your prediction down, then run compare.py. Remember that the video ran each attack only once.

    A hint

    Harm means a note was deleted, an email was sent, or the vault was read.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six attacks I chose, one local model (llama3.2), one seed and one run each, on made-up notes in a temporary folder; says nothing about attacks that are not on the list.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos