learning-it by example

#96 AI security and governance 5 min

Content Moderation With a Small Open Model vs a Word List

My banned-word list blocked a made-up test comment about killing a Python process.

1. Content moderation is described as two mistakes in one job. Which two?
2. On twenty made-up comments labelled by the author, 13 should be blocked and 7 are safe. How did a 12-word banned list and llama3.2 (one seed, a label-only schema) score?
3. In a test of llama3.2 against a banned-word list on twenty comments the author wrote, why might the model's result look better than it deserves?
4. In a moderation script that asks llama3.2 for one label, how does the code keep the model's answer easy to read?
5. A 12-word banned list blocked 9 of 13 unsafe made-up comments, and one of those nine catches was luck. Why?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Remove one banned word

    Copy the code folder first. In judge.py remove 'kill' from BANNED. In a Python session in that folder run: from messages import MESSAGES and from judge import by_words. Count how many comments labelled spam or abuse get 'unsafe' from by_words, and how many safe ones do. Predict both numbers first. With the original list you should get 9 and 4, and with kill removed 8 and 2.

    A hint

    Build two lists from MESSAGES, one where the label is not safe and one where it is safe, and count the 'unsafe' answers in each.

  2. Remove three words

    Remove 'kill', 'destroy' and 'stupid' from BANNED, then count the two numbers again as in the previous task, in a fresh Python session (the open one still holds the old list). Predict first. You should see the list catch 8 of the 13 and wrongly block 0 of the 7. Then add 'subscribe' to BANNED and count again, which gives 9 of 13 and still 0 of 7.

    A hint

    Ask yourself which comments the model got right that this list still cannot, and what it would take to keep the second number at 0.

  3. Find the misses of the list

    Put kill, destroy and stupid back and remove subscribe, so that BANNED is the original list, and start a fresh Python session. Then print every comment labelled spam or abuse that by_words calls 'safe'. Predict how many there are before you run it, because the video says the list let through four of the thirteen. Then decide, for each one, which word you would have to add, and whether that word would block any of the safe comments.

    A hint

    Look at the comment written with numbers in place of letters, and the one that says to delete the channel.

  4. Add a comment of your own

    This needs Ollama with llama3.2, so treat it as a prediction. Add a tricky made-up comment and a label you choose to the end of MESSAGES in messages.py, for example a safe comment that uses a rude word about a thing. Predict whether the banned words and the small model will each get it right, then run compare.py and compare.

    A hint

    The video says to keep a list of the comments you got wrong and to count both kinds of mistake every time.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Twenty comments I wrote and labelled myself, in English and mild; one seed, one small local model (llama3.2). The prompt was written after seeing the word list's mistakes, so the model had a head start. One of the word list's nine catches is a substring accident (skills contains kill). No dedicated safeguard model such as Llama Guard was tested.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos