#96 AI security and governance 5 min
Content Moderation With a Small Open Model vs a Word List
My banned-word list blocked a made-up test comment about killing a Python process.
Remove one banned word
Copy the code folder first. In judge.py remove 'kill' from BANNED. In a Python session in that folder run: from messages import MESSAGES and from judge import by_words. Count how many comments labelled spam or abuse get 'unsafe' from by_words, and how many safe ones do. Predict both numbers first. With the original list you should get 9 and 4, and with kill removed 8 and 2.
A hint
Build two lists from MESSAGES, one where the label is not safe and one where it is safe, and count the 'unsafe' answers in each.
Remove three words
Remove 'kill', 'destroy' and 'stupid' from BANNED, then count the two numbers again as in the previous task, in a fresh Python session (the open one still holds the old list). Predict first. You should see the list catch 8 of the 13 and wrongly block 0 of the 7. Then add 'subscribe' to BANNED and count again, which gives 9 of 13 and still 0 of 7.
A hint
Ask yourself which comments the model got right that this list still cannot, and what it would take to keep the second number at 0.
Find the misses of the list
Put kill, destroy and stupid back and remove subscribe, so that BANNED is the original list, and start a fresh Python session. Then print every comment labelled spam or abuse that by_words calls 'safe'. Predict how many there are before you run it, because the video says the list let through four of the thirteen. Then decide, for each one, which word you would have to add, and whether that word would block any of the safe comments.
A hint
Look at the comment written with numbers in place of letters, and the one that says to delete the channel.
Add a comment of your own
This needs Ollama with llama3.2, so treat it as a prediction. Add a tricky made-up comment and a label you choose to the end of MESSAGES in messages.py, for example a safe comment that uses a rude word about a thing. Predict whether the banned words and the small model will each get it right, then run compare.py and compare.
A hint
The video says to keep a list of the comments you got wrong and to count both kinds of mistake every time.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Twenty comments I wrote and labelled myself, in English and mild; one seed, one small local model (llama3.2). The prompt was written after seeing the word list's mistakes, so the model had a head start. One of the word list's nine catches is a substring accident (skills contains kill). No dedicated safeguard model such as Llama Guard was tested.
Up next in AI security and governance
More in AI security and governance
- #18 Prompt Injection Live: Attacking My Own LLM App
- #19 Redact Personal Data Before Your LLM Sees It (Python)
- #88 Indirect Prompt Injection: A Note That Gave Orders
- #89 Jailbreak-Test Your Own App: Eight Wrappers, Counted
- #90 Stop Leaking API Keys: Build a Secret Scanner in Python
- #91 Rate Limiting an LLM API: Token Bucket in FastAPI
