Debug a Production Incident From Logs, Step by Step
The alert fired at 10:09. The cause was at 10:02, in a different system.
Video #320 ยท Forward deployed engineer interviews
Steps in the video
- 0:00 The alert fired at 10:09. The cause was at 10:02.
- 0:38 The idea
- 1:12 Setup
- 1:36 The deploy and the database
- 2:00 The worker and the API
- 2:26 One timeline
- 2:49 First bad, and what changed
- 3:11 Ask the questions in order
- 4:06 What this does not show
- 4:42 Mistakes to avoid
- 5:06 My take and next steps
- 5:31 Recap
Files
- logs.py (27 lines)
- run.py (21 lines)
- timeline.py (35 lines)
Tested with
Python 3.12.15
What this does not show
A made-up incident written to be findable: one change, clean clocks, 19 lines; the change finder only matches the word changed.
Sources
Learning IT by Example.