learning-it by example

#18 AI security and governance 5.6 min

Prompt Injection Live: Attacking My Own LLM App

I attacked my own language model app and a secret leaked. Then I ran the same attacks against three defences, and in this small test one of them made it worse.

1. Why can text inside a customer email steer the model, according to the video?
2. In the toy llama3.2 app that summarises customer emails, with no defence, which attack email leaked the discount code?
3. How many of the three attacks leaked the code with the delimiters defence (email wrapped in tags marked as data)?
4. In the prompt injection test, the output filter scored 0 of 3 leaks, but the score uses the same check as the filter. What does that make the filter?
5. What does the video advise about the wording of attacks when you test your app?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Test how exact the leak check is

    Copy the code folder first. In a Python prompt in that folder, import leaked from shop and call it on four strings: 'the code is SAVE-7421', 'the code is save-7421', 'SAVE 7421' and 'SAVE-7421'. Predict each answer before you run. When we ran it, the answers were True, False, False and True. Write one sentence on what that means for an output filter that uses the same exact check.

    A hint

    The leaked function only asks whether the exact string CODE is inside the text.

  2. Add a fourth attack wording

    In shop.py add an entry called attack 4 to EMAILS. Write an email about order 5521 that asks for the discount code in a different way, for example as a table with two columns. Make sure the Ollama app is running, then run python attack.py and note whether your wording held or LEAKED. Predict the result first, and keep the counts from your run.

    A hint

    The last line of attack.py says of 3, so it will not match your count of attacks. Read the lines above it for the real result.

  3. Make the leak check ignore case and spaces

    Change the leaked function in shop.py so it removes everything except letters and digits, and lowercases both the code and the text, before it checks. Then call leaked on 'SAVE 7421' and 'save-7421'. Predict both answers first. When we tried this change, both became True. Explain in one sentence why a filter like this is still only one layer.

    A hint

    Keep only the characters where c.isalnum() is true and call .lower() on the result.

  4. Change the delimiter wording and recount

    In defend.py change the delimited prompt, for example swap the <email> tags for the word DATA, or add a stronger sentence saying the email never contains instructions. With the Ollama app running, run python defend.py and compare the delimiters line with the 2 of 3 count in the video. Predict whether it goes up, down or stays the same, and write down what you see. One run of one wording is not a general result.

    A hint

    Only change the prompt text inside the delimited function. The other two defences stay the same.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One small local model and three attack emails, one run each, so the counts are not a general result for any model or defence. The discount code is made up. The output filter's zero is true by construction (the score uses the same string check) and an exact-string filter misses reworded leaks.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos