LLM Failures: Timeouts, Retries and Fallbacks
In a test with injected failures, retries fixed a simulated busy server, then made twelve attempts and fixed nothing.
Video #60 ยท LLM engineering and evals
Steps in the video
Files
- compare.py (25 lines)
- flaky.py (25 lines)
- policies.py (36 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Four requests per case, one seed and one small local model (llama3.2). The busy and down servers are faults injected by the code; only the short timeout is a real httpx timeout. The backoff has no jitter, the waits are far shorter than production values, and seconds were not measured.
Sources
Learning IT by Example.