learning-it by example

#320 Forward deployed engineer interviews 6 min

Debug a Production Incident From Logs, Step by Step

The alert fired at 10:09. The cause was at 10:02, in a different system.

1. In the made-up incident, the alert fired at 10:09 on the API. Where did the video say to start debugging?
2. In the merged-logs incident demo, what was the first bad event, and how long before the alert did it happen?
3. What does the video say about real incidents compared with its made-up one?
4. How does the changed_before function find a change?
5. Before working out the cause in a real incident, what does the video say to do first?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Hide the word 'changed'

    Copy the code folder first. In logs.py, in the DEPLOY text, change 'config changed:' to 'config updated:'. Predict which lines of the output disappear, then run python run.py and compare it with the original output. The heading 'from the change to the alert' is then stale. Then change 'config updated:' back to 'config changed:', because the next exercise needs it.

    A hint

    Think about what the changed_before function searches for. The timeline at the bottom uses the same word.

  2. Shorten the change window

    In timeline.py change window=300 to window=60 in the changed_before function. The release was 71 seconds before the first bad event. Predict whether the 'changed 71 s earlier' lines still appear, and whether the deploy line still appears in the timeline at the bottom. Run python run.py to check. Then change window=60 back to window=300.

    A hint

    The window and the final timeline are chosen by different code. In my run the 'changed ... earlier' lines disappeared but the timeline still showed the deploy line.

  3. Add an earlier warning

    In logs.py add a new first line to the WORKER text, followed by a line break so that the old first line stays on its own line: 10:01:30 worker WARN a retry was needed. Predict what the 'first bad event' line will say, and whether any change will be found before it. Run python run.py. Then write one sentence on why the first bad event is not always the cause.

    A hint

    The release is at 10:02:30, after the new line. I saw '20 lines merged', a first bad event at 10:01:30 and no change listed.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A made-up incident written to be findable: one change, clean clocks, 19 lines; the change finder only matches the word changed.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos