#99 AI security and governance 5.3 min
Pickle vs Safetensors: Why Loading a Model Can Run Code
Loading a pickle file, a format model files can use, ran code I had hidden inside it, four files out of four.
Switch off the bad-module list
Copy the code folder first. In guards.py remove 'os', 'posix' and 'nt' from the DENY set, then run compare.py. Predict what the bad-module line says. You should see the bad-module list flag 0 of 4 files, because the folder creation file was the only one it caught before.
A hint
Keep subprocess and socket in the set. Only the three operating system names come out.
Allow one more global
In guards.py add ('__builtin__', 'print') to the ALLOW set. Pickle protocol 2 uses the older name __builtin__ for the built-in functions. Predict how many of the four files the allow list stops now, and which one gets through. You should see 3 of 4 stopped, and an extra line saying loaded, which is the print file running.
A hint
The video warns that each name you add to an allow list is a risk.
Allow the file copy
Remove the previous change, and instead add ('shutil', 'copyfile') to ALLOW. Predict how many files the allow list stops, and what happens to the file copy attack. You should see the allow list stop 3 of 4; the copy attack ran too, and made its copy of seed.txt inside the temporary folder that compare.py creates, so the count is the only sign you will see. Do this only in a scratch copy of the folder.
A hint
Think about why the video says to be careful adding names until things stop failing.
Say why safetensors refused them
Read compare.py and guards.py again, then write two sentences in your own words. The first should say what the allow list lets through and why it refuses the rest. The second should say what kind of data safetensors stores and why that makes it a different idea from a guard. Check your answer against the recap of the video.
A hint
One guard filters code the file wants to run. The other format has no place to put code.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Four harmless payloads written by the author. The bad-module scan only reads the older pickle format used here (protocol 2); files saved with the default protocol show it no module names. A real model file may need more allow-list globals. Safetensors holds tensors, not everything in a model. No scanner service and no real model file were tested.
Up next in AI security and governance
More in AI security and governance
- #18 Prompt Injection Live: Attacking My Own LLM App
- #19 Redact Personal Data Before Your LLM Sees It (Python)
- #88 Indirect Prompt Injection: A Note That Gave Orders
- #89 Jailbreak-Test Your Own App: Eight Wrappers, Counted
- #90 Stop Leaking API Keys: Build a Secret Scanner in Python
- #91 Rate Limiting an LLM API: Token Bucket in FastAPI
Related videos
- #18 Prompt Injection Live: Attacking My Own LLM App AI security and governance
- #88 Indirect Prompt Injection: A Note That Gave Orders AI security and governance
- #93 OWASP Top 10 for LLM Apps: Three Risks Tested in Code AI security and governance
- #94 Can Your Model Leak Its System Prompt? I Tried 8 Ways AI security and governance
