learning-it by example

#319 Forward deployed engineer interviews 6.2 min

Triage LLM Failures From Logs: Find the Top Cause

The biggest cause of failures was hiding in a bucket called 'other'.

1. In the LLM error-log triage demo, the biggest bucket after the first three rules was called 'other'. What was the right response?
2. After the timeout rule was fixed, how many of the 150 failures were left in 'other', and which cause was biggest?
3. What does the video say a count of failures leaves out?
4. Why did the first timeout rule, the pattern 'time[d ]?out', miss lines such as 'read timed out'?
5. After the fix, what share of all 150 failures did the top three causes (timeout, bad JSON and rate limit) cover together?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Remove the refusal rule

    Copy the code folder first. In triage.py delete the line that holds the 'refusal' rule from MORE_RULES. Predict how many lines will be left in 'other' after the timeout rule is fixed, and which kind of failure they will be. Run python run.py and read the 'still unsorted' lines too. The output label 'eight rules' is now stale, because only seven rules are left. Then put the refusal line back in MORE_RULES, because the next exercise needs it.

    A hint

    A rule you delete does not make its failures go away, they fall into 'other'. I saw 8 left after the fix.

  2. See the Pareto before the fix

    At the end of run.py add a second Pareto, like the one already there but built with count(lines, FIRST_RULES + MORE_RULES) instead of FIXED_RULES, with the heading 'the Pareto before the fix'. Predict which bucket comes second. Run python run.py and compare it with the Pareto after the fix.

    A hint

    Reuse the same for loop and print line. In my run bad JSON came first, 'other' with 28 came second, and timeout showed only 10.

  3. Change the random seed

    In logs.py change make(seed=9) to make(seed=10). Predict whether the top three causes will change, and whether the number left in 'other' after three rules will change. Run python run.py and compare it with the original output.

    A hint

    The kinds and their counts are fixed in KINDS. Only the choice of wording and the order are random. I saw 79 left after three rules.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

150 generated log lines in proportions chosen by hand, with a planted rule bug; regex rules, first match wins; count is not harm.

Up next in Forward deployed engineer interviews

More in Forward deployed engineer interviews

All Forward deployed engineer interviews videos

Related videos