learning-it by example

Back to the video

Triage LLM Failures From Logs: Find the Top Cause

The biggest cause of failures was hiding in a bucket called 'other'.

Video #319 ยท Forward deployed engineer interviews

Steps in the video

  1. 0:00 The biggest cause of failures was hiding in a bucket called other
  2. 0:33 The idea
  3. 1:09 Setup
  4. 1:33 The failures
  5. 2:05 The rules
  6. 2:39 Bucket and count
  7. 2:55 Read what is left
  8. 3:12 The Pareto
  9. 3:28 Three passes
  10. 4:21 What this does not show
  11. 4:54 Mistakes to avoid
  12. 5:14 My take and next steps
  13. 5:42 Recap

Files

  • logs.py (26 lines)
  • run.py (18 lines)
  • triage.py (48 lines)

Download the code (zip)

Tested with

Python 3.12.15

What this does not show

150 generated log lines in proportions chosen by hand, with a planted rule bug; regex rules, first match wins; count is not harm.

Sources

Learning IT by Example.