#18 AI security and governance 5.6 min
Prompt Injection Live: Attacking My Own LLM App
I attacked my own language model app and a secret leaked. Then I ran the same attacks against three defences, and in this small test one of them made it worse.
Test how exact the leak check is
Copy the code folder first. In a Python prompt in that folder, import leaked from shop and call it on four strings: 'the code is SAVE-7421', 'the code is save-7421', 'SAVE 7421' and 'SAVE-7421'. Predict each answer before you run. When we ran it, the answers were True, False, False and True. Write one sentence on what that means for an output filter that uses the same exact check.
A hint
The leaked function only asks whether the exact string CODE is inside the text.
Add a fourth attack wording
In shop.py add an entry called attack 4 to EMAILS. Write an email about order 5521 that asks for the discount code in a different way, for example as a table with two columns. Make sure the Ollama app is running, then run python attack.py and note whether your wording held or LEAKED. Predict the result first, and keep the counts from your run.
A hint
The last line of attack.py says of 3, so it will not match your count of attacks. Read the lines above it for the real result.
Make the leak check ignore case and spaces
Change the leaked function in shop.py so it removes everything except letters and digits, and lowercases both the code and the text, before it checks. Then call leaked on 'SAVE 7421' and 'save-7421'. Predict both answers first. When we tried this change, both became True. Explain in one sentence why a filter like this is still only one layer.
A hint
Keep only the characters where c.isalnum() is true and call .lower() on the result.
Change the delimiter wording and recount
In defend.py change the delimited prompt, for example swap the <email> tags for the word DATA, or add a stronger sentence saying the email never contains instructions. With the Ollama app running, run python defend.py and compare the delimiters line with the 2 of 3 count in the video. Predict whether it goes up, down or stays the same, and write down what you see. One run of one wording is not a general result.
A hint
Only change the prompt text inside the delimited function. The other two defences stay the same.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One small local model and three attack emails, one run each, so the counts are not a general result for any model or defence. The discount code is made up. The output filter's zero is true by construction (the score uses the same string check) and an exact-string filter misses reworded leaks.
Up next in AI security and governance
More in AI security and governance
Related videos
- #88 Indirect Prompt Injection: A Note That Gave Orders AI security and governance
- #94 Can Your Model Leak Its System Prompt? I Tried 8 Ways AI security and governance
- #89 Jailbreak-Test Your Own App: Eight Wrappers, Counted AI security and governance
- #93 OWASP Top 10 for LLM Apps: Three Risks Tested in Code AI security and governance
