learning-it by example

#99 AI security and governance 5.3 min

Pickle vs Safetensors: Why Loading a Model Can Run Code

Loading a pickle file, a format model files can use, ran code I had hidden inside it, four files out of four.

1. Why can loading a pickle file from someone else run their code?
2. Four harmless booby-trapped pickle files (saved with pickle protocol 2) were loaded. What did plain pickle, a bad-module list, an allow list and safetensors each do?
3. Why does a list of bad modules not make loading a pickle file safe?
4. How does the allow-list guard from the Python documentation work in the video?
5. What does safetensors store, and what is a limit of it that the video states?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Switch off the bad-module list

    Copy the code folder first. In guards.py remove 'os', 'posix' and 'nt' from the DENY set, then run compare.py. Predict what the bad-module line says. You should see the bad-module list flag 0 of 4 files, because the folder creation file was the only one it caught before.

    A hint

    Keep subprocess and socket in the set. Only the three operating system names come out.

  2. Allow one more global

    In guards.py add ('__builtin__', 'print') to the ALLOW set. Pickle protocol 2 uses the older name __builtin__ for the built-in functions. Predict how many of the four files the allow list stops now, and which one gets through. You should see 3 of 4 stopped, and an extra line saying loaded, which is the print file running.

    A hint

    The video warns that each name you add to an allow list is a risk.

  3. Allow the file copy

    Remove the previous change, and instead add ('shutil', 'copyfile') to ALLOW. Predict how many files the allow list stops, and what happens to the file copy attack. You should see the allow list stop 3 of 4; the copy attack ran too, and made its copy of seed.txt inside the temporary folder that compare.py creates, so the count is the only sign you will see. Do this only in a scratch copy of the folder.

    A hint

    Think about why the video says to be careful adding names until things stop failing.

  4. Say why safetensors refused them

    Read compare.py and guards.py again, then write two sentences in your own words. The first should say what the allow list lets through and why it refuses the rest. The second should say what kind of data safetensors stores and why that makes it a different idea from a guard. Check your answer against the recap of the video.

    A hint

    One guard filters code the file wants to run. The other format has no place to put code.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Four harmless payloads written by the author. The bad-module scan only reads the older pickle format used here (protocol 2); files saved with the default protocol show it no module names. A real model file may need more allow-list globals. Safetensors holds tensors, not everything in a model. No scanner service and no real model file were tested.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos