learning-it by example

Back to the video

LLM Failures: Timeouts, Retries and Fallbacks

In a test with injected failures, retries fixed a simulated busy server, then made twelve attempts and fixed nothing.

Video #60 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Retries helped with one failure of three, in a test
  2. 0:27 The idea
  3. 1:05 Setup
  4. 1:27 A call that can fail
  5. 1:54 Four policies
  6. 2:16 Three failures
  7. 3:18 What this does not show
  8. 3:48 Mistakes to avoid
  9. 4:12 My take and next steps
  10. 4:35 Recap

Files

  • compare.py (25 lines)
  • flaky.py (25 lines)
  • policies.py (36 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Four requests per case, one seed and one small local model (llama3.2). The busy and down servers are faults injected by the code; only the short timeout is a real httpx timeout. The backoff has no jitter, the waits are far shorter than production values, and seconds were not measured.

Sources

Learning IT by Example.