learning-it by example

#315 Forward deployed engineer interviews 5.3 min

Safe Rollouts: Canary and Rollback, Simulated

The same bug reached 100, 30 or 830 requests. The difference was a canary that someone watched.

1. What is a canary release?
2. In the simulation of 1,000 requests with a buggy version 2, how many requests reached version 2 with each way of rolling out?
3. In the code, when does the check roll everyone back to version 1?
4. Why did the canary that nobody watched never roll back in the simulation?
5. Which limit of the simulation does the video state about small bugs?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the three rollouts

    Copy the code folder first. Run python run.py. Read the three rows and write down, for each way of rolling out, how many requests reached version 2 and whether it rolled back. You should see 100, 30 and 830 requests, and the last one never rolling back.

    A hint

    The column called on v2 is the exposure.

  2. Check more often

    In rollout.py change watch=100 to watch=50 in the run function. Predict how the exposure of the all at once and the watched canary rows will change, then run python run.py. With this change all at once exposes 50 requests and the watched canary 22, while the unwatched canary reaches 922.

    A hint

    The check can only act every watch requests.

  3. Check less often

    In rollout.py change watch=100 to watch=250. Predict the all at once exposure, then run python run.py. With this change all at once exposes 250 requests before it rolls back, and the watched canary 17. Why is checking too rarely on the video's list of mistakes?

    A hint

    All at once cannot roll back before the first check.

  4. Make the bug small

    In rollout.py change the FAILS value for version 2 from 0.125 to 0.01. This bug now fails 1 in 100 requests, the same as version 1. Predict whether the watched canary will roll back, then run python run.py. With this change none of the three roll back, and the watched canary reaches 830 requests. What does this show about the 4 percent limit?

    A hint

    Compare the error rate of the bug with the limit of 0.04.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Simulated traffic, made-up failure rates, one seed. In a side check the watched canary's 30 ranged from 10 to 143 over 1,000 seeds; 830 follows from the plan. Version 1's failures are not simulated. A bug smaller than the 4 percent limit is not caught. Real canaries watch more than errors.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos