learning-it by example

Back to the video

Debug a Production Incident From Logs, Step by Step

The alert fired at 10:09. The cause was at 10:02, in a different system.

Video #320 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 The alert fired at 10:09. The cause was at 10:02.
  2. 0:38 The idea
  3. 1:12 Setup
  4. 1:36 The deploy and the database
  5. 2:00 The worker and the API
  6. 2:26 One timeline
  7. 2:49 First bad, and what changed
  8. 3:11 Ask the questions in order
  9. 4:06 What this does not show
  10. 4:42 Mistakes to avoid
  11. 5:06 My take and next steps
  12. 5:31 Recap

Files

  • logs.py (27 lines)
  • run.py (21 lines)
  • timeline.py (35 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

A made-up incident written to be findable: one change, clean clocks, 19 lines; the change finder only matches the word changed.

Sources

Learning IT by Example.