#60 LLM engineering and evals 5 min
LLM Failures: Timeouts, Retries and Fallbacks
In a test with injected failures, retries fixed a simulated busy server, then made twelve attempts and fixed nothing.
Read the four policies
Copy the code folder first. Open policies.py and write down, for plain, retry, backoff and fallback, how many times each calls the model at most for one request, and which of them waits between attempts. Then work out the longest total wait backoff could add for one request with the default settings.
A hint
The waits are wait times 2 to the power n, but only between attempts.
Make the busy server busier
In compare.py change fail_rate=0.4 to fail_rate=0.7 in the busy server condition. Predict how many of the 4 requests plain and retry x3 will answer, and how many attempts retry x3 will make. Then run python compare.py. Write your prediction first. Then check it: with the same seed, retry x3 should answer 3 of 4 with 8 attempts, and retry + fallback should give 3 model answers and 1 fallback. Then change fail_rate back to 0.4.
A hint
The busy server fault is made by a seeded random number, so the counts repeat on every run.
Switch the faults off
In the busy server condition set fail_rate to 0.0 (the original file has 0.4, so undo the 0.7 edit first if you did the last exercise) and run python compare.py. Predict what the four policies will show for answers and attempts, and why retrying costs nothing when nothing fails. Compare with your prediction. For the plain policy you should see model 4 and attempts 4. Then set fail_rate back to 0.4.
A hint
No attempt raises an error, so no retry ever happens.
Choose a safer timeout
In compare.py the timeout condition uses 0.3 seconds. Predict whether a timeout of 60 seconds, the default in flaky.py, would make the fallback unnecessary for the same four requests. If Ollama and llama3.2 are installed, change it (the printed condition name will still say timeout too short (0.3s), so read it as 60 seconds) and run python compare.py, then write one sentence on how you would pick a timeout from a normal reply.
A hint
The video said to set the timeout from a measured normal reply.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Four requests per case, one seed and one small local model (llama3.2). The busy and down servers are faults injected by the code; only the short timeout is a real httpx timeout. The backoff has no jitter, the waits are far shorter than production values, and seconds were not measured.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
