learning-it by example

Back to the video

Safe Rollouts: Canary and Rollback, Simulated

The same bug reached 100, 30 or 830 requests. The difference was a canary that someone watched.

Video #315 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 The buggy version took 830 requests because nobody was watching
  2. 0:37 The idea
  3. 1:14 Setup
  4. 1:38 The rollout
  5. 2:05 Three ways to roll out
  6. 3:00 What this does not show
  7. 3:54 Mistakes to avoid
  8. 4:17 My take and next steps
  9. 4:49 Recap

Files

  • rollout.py (28 lines)
  • run.py (13 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

Simulated traffic, made-up failure rates, one seed. In a side check the watched canary's 30 ranged from 10 to 143 over 1,000 seeds; 830 follows from the plan. Version 1's failures are not simulated. A bug smaller than the 4 percent limit is not caught. Real canaries watch more than errors.

Sources

Learning IT by Example.