#319 Forward deployed engineer interviews 6.2 min
Triage LLM Failures From Logs: Find the Top Cause
The biggest cause of failures was hiding in a bucket called 'other'.
Remove the refusal rule
Copy the code folder first. In triage.py delete the line that holds the 'refusal' rule from MORE_RULES. Predict how many lines will be left in 'other' after the timeout rule is fixed, and which kind of failure they will be. Run python run.py and read the 'still unsorted' lines too. The output label 'eight rules' is now stale, because only seven rules are left. Then put the refusal line back in MORE_RULES, because the next exercise needs it.
A hint
A rule you delete does not make its failures go away, they fall into 'other'. I saw 8 left after the fix.
See the Pareto before the fix
At the end of run.py add a second Pareto, like the one already there but built with count(lines, FIRST_RULES + MORE_RULES) instead of FIXED_RULES, with the heading 'the Pareto before the fix'. Predict which bucket comes second. Run python run.py and compare it with the Pareto after the fix.
A hint
Reuse the same for loop and print line. In my run bad JSON came first, 'other' with 28 came second, and timeout showed only 10.
Change the random seed
In logs.py change make(seed=9) to make(seed=10). Predict whether the top three causes will change, and whether the number left in 'other' after three rules will change. Run python run.py and compare it with the original output.
A hint
The kinds and their counts are fixed in KINDS. Only the choice of wording and the order are random. I saw 79 left after three rules.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
150 generated log lines in proportions chosen by hand, with a planted rule bug; regex rules, first match wins; count is not harm.
Up next in Forward deployed engineer interviews
More in Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested
- #303 The FDE Decomposition Round: Vague Ask to a Plan
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule
- #305 Retry a Flaky API Without Duplicates: Idempotency Keys
- #306 Webhooks Done Right: Signatures, Retries and Replays
- #307 Schema Drift: Map a Customer's Data and Catch Changes
Related videos
- #320 Debug a Production Incident From Logs, Step by Step Forward deployed engineer interviews
- #302 Forward Deployed Engineer Interview: What Is Tested Forward deployed engineer interviews
- #303 The FDE Decomposition Round: Vague Ask to a Plan Forward deployed engineer interviews
- #304 Scope an AI Pilot: Metric, Baseline and Stop Rule Forward deployed engineer interviews
