learning-it by example

#101 AI security and governance 5.4 min

Differential Privacy Basics: Noisy Counts in Python

A made-up count of 412 told a simulated attacker whether one person was in the data.

1. How large is the noise that the video adds to a count, and why that size?
2. In the attacker game with one noisy answer, how often was the attacker right at epsilon 1, and at epsilon 0.1?
3. What happened when the attacker asked the same question at epsilon 1 twenty times and averaged the answers?
4. How does the code make the Laplace noise?
5. What kind of attacker does the privacy test assume, which the video calls the strongest case?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Try epsilon 0.5

    Copy the code folder first. In compare.py change the loop for epsilon in (10, 1, 0.1) to for epsilon in (0.5,) and run it. Predict the typical error and how often the attacker is right, then compare with the lines for 1 and 0.1 in the video. When I ran it the typical error was 2.0 and the attacker was right 62% of the time.

    A hint

    Epsilon 0.5 sits between epsilon 1 and epsilon 0.1.

  2. Ask five times instead of twenty

    In compare.py change guess_right(1, 20) to guess_right(1, 5) and the words in the print text to match. Predict whether the attacker does better or worse than with 20 questions and than with 1 question. When I ran it the attacker was right 80% of the time at epsilon 1 with 5 questions, between 70% for one question and 94% for twenty.

    A hint

    More questions means more fresh noise to average away.

  3. Change the seed

    In compare.py change random.Random(7) to random.Random(8) and run it. Predict which lines change. When I ran it only the typical error at epsilon 0.1 changed, from 9.9 to 10.2, and the percentages stayed the same. Then explain why the video says the seed makes the numbers repeat, but a real system needs a carefully built noise source.

    A hint

    The numbers in the video repeat only on the same Python version with the same seed.

  4. Spend the budget on a tiny epsilon

    Change guess_right(1, 20) in the last print to guess_right(0.1, 20), leaving the printed words alone, and run it. Predict whether twenty averaged answers at epsilon 0.1 help the attacker as much as at epsilon 1. When I ran it the attacker was right 56% of the time. Compare it with 52% for one answer at epsilon 0.1.

    A hint

    A smaller epsilon means much more noise in every answer.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One made-up count and one made-up person; a seeded teaching simulation in plain Python against an attacker who knows both possible counts; not noise you should use in production.

Up next in AI security and governance

More in AI security and governance

All AI security and governance videos

Related videos