Triage LLM Failures From Logs: Find the Top Cause
The biggest cause of failures was hiding in a bucket called 'other'.
Video #319 ยท Forward deployed engineer interviews
Steps in the video
Files
- logs.py (26 lines)
- run.py (18 lines)
- triage.py (48 lines)
Tested with
Python 3.12.15
What this does not show
150 generated log lines in proportions chosen by hand, with a planted rule bug; regex rules, first match wins; count is not harm.
Sources
- https://alexstrick.com/posts/2025-05-23-error-analysis-to-find-failure-modes.html
- https://claveprep.com/blog/forward-deployed-engineer-system-design-interview
- https://hamel.dev/blog/posts/evals-faq/why-is-error-analysis-so-important-in-llm-evals-and-how-is-it-performed.html
- https://langfuse.com/blog/2025-08-29-error-analysis-to-evaluate-llm-applications
Learning IT by Example.