#315 Forward deployed engineer interviews 5.3 min
Safe Rollouts: Canary and Rollback, Simulated
The same bug reached 100, 30 or 830 requests. The difference was a canary that someone watched.
Run the three rollouts
Copy the code folder first. Run python run.py. Read the three rows and write down, for each way of rolling out, how many requests reached version 2 and whether it rolled back. You should see 100, 30 and 830 requests, and the last one never rolling back.
A hint
The column called on v2 is the exposure.
Check more often
In rollout.py change watch=100 to watch=50 in the run function. Predict how the exposure of the all at once and the watched canary rows will change, then run python run.py. With this change all at once exposes 50 requests and the watched canary 22, while the unwatched canary reaches 922.
A hint
The check can only act every watch requests.
Check less often
In rollout.py change watch=100 to watch=250. Predict the all at once exposure, then run python run.py. With this change all at once exposes 250 requests before it rolls back, and the watched canary 17. Why is checking too rarely on the video's list of mistakes?
A hint
All at once cannot roll back before the first check.
Make the bug small
In rollout.py change the FAILS value for version 2 from 0.125 to 0.01. This bug now fails 1 in 100 requests, the same as version 1. Predict whether the watched canary will roll back, then run python run.py. With this change none of the three roll back, and the watched canary reaches 830 requests. What does this show about the 4 percent limit?
A hint
Compare the error rate of the bug with the limit of 0.04.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Simulated traffic, made-up failure rates, one seed. In a side check the watched canary's 30 ranged from 10 to 143 over 1,000 seeds; 830 follows from the plan. Version 1's failures are not simulated. A bug smaller than the 4 percent limit is not caught. Real canaries watch more than errors.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #302 Forward Deployed Engineer Interview: What Is Tested Forward deployed engineer interviews
- #303 The FDE Decomposition Round: Vague Ask to a Plan Forward deployed engineer interviews
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys Forward deployed engineer interviews
- #306 Webhooks Done Right: Signatures, Retries and Replays Forward deployed engineer interviews
