#320 Forward deployed engineer interviews 6 min
Debug a Production Incident From Logs, Step by Step
The alert fired at 10:09. The cause was at 10:02, in a different system.
Hide the word 'changed'
Copy the code folder first. In logs.py, in the DEPLOY text, change 'config changed:' to 'config updated:'. Predict which lines of the output disappear, then run python run.py and compare it with the original output. The heading 'from the change to the alert' is then stale. Then change 'config updated:' back to 'config changed:', because the next exercise needs it.
A hint
Think about what the changed_before function searches for. The timeline at the bottom uses the same word.
Shorten the change window
In timeline.py change window=300 to window=60 in the changed_before function. The release was 71 seconds before the first bad event. Predict whether the 'changed 71 s earlier' lines still appear, and whether the deploy line still appears in the timeline at the bottom. Run python run.py to check. Then change window=60 back to window=300.
A hint
The window and the final timeline are chosen by different code. In my run the 'changed ... earlier' lines disappeared but the timeline still showed the deploy line.
Add an earlier warning
In logs.py add a new first line to the WORKER text, followed by a line break so that the old first line stays on its own line: 10:01:30 worker WARN a retry was needed. Predict what the 'first bad event' line will say, and whether any change will be found before it. Run python run.py. Then write one sentence on why the first bad event is not always the cause.
A hint
The release is at 10:02:30, after the new line. I saw '20 lines merged', a first bad event at 10:01:30 and no change listed.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
A made-up incident written to be findable: one change, clean clocks, 19 lines; the change finder only matches the word changed.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #319 Triage LLM Failures From Logs: Find the Top Cause Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested Forward deployed engineer interviews
- #303 The FDE Decomposition Round: Vague Ask to a Plan Forward deployed engineer interviews
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule Forward deployed engineer interviews
