learning-it by example

#60 LLM engineering and evals 5 min

LLM Failures: Timeouts, Retries and Fallbacks

In a test with injected failures, retries fixed a simulated busy server, then made twelve attempts and fixed nothing.

1. In the video's test of four retry policies (llama3.2, four requests each), a model call fails because the timeout is 0.3 seconds, shorter than the model needs. What did retries do?
2. In the video's simulated failure test (llama3.2, four requests, a seeded random number making 40 percent of attempts fail), how did plain calls and retry x3 compare?
3. What is a fallback in the video's retry and fallback policy, and what is its risk?
4. What is backoff in the retry policies, and what is jitter?
5. In the video's simulated failure test, how does the code make the same attempts fail every time it is run on the busy server?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Read the four policies

    Copy the code folder first. Open policies.py and write down, for plain, retry, backoff and fallback, how many times each calls the model at most for one request, and which of them waits between attempts. Then work out the longest total wait backoff could add for one request with the default settings.

    A hint

    The waits are wait times 2 to the power n, but only between attempts.

  2. Make the busy server busier

    In compare.py change fail_rate=0.4 to fail_rate=0.7 in the busy server condition. Predict how many of the 4 requests plain and retry x3 will answer, and how many attempts retry x3 will make. Then run python compare.py. Write your prediction first. Then check it: with the same seed, retry x3 should answer 3 of 4 with 8 attempts, and retry + fallback should give 3 model answers and 1 fallback. Then change fail_rate back to 0.4.

    A hint

    The busy server fault is made by a seeded random number, so the counts repeat on every run.

  3. Switch the faults off

    In the busy server condition set fail_rate to 0.0 (the original file has 0.4, so undo the 0.7 edit first if you did the last exercise) and run python compare.py. Predict what the four policies will show for answers and attempts, and why retrying costs nothing when nothing fails. Compare with your prediction. For the plain policy you should see model 4 and attempts 4. Then set fail_rate back to 0.4.

    A hint

    No attempt raises an error, so no retry ever happens.

  4. Choose a safer timeout

    In compare.py the timeout condition uses 0.3 seconds. Predict whether a timeout of 60 seconds, the default in flaky.py, would make the fallback unnecessary for the same four requests. If Ollama and llama3.2 are installed, change it (the printed condition name will still say timeout too short (0.3s), so read it as 60 seconds) and run python compare.py, then write one sentence on how you would pick a timeout from a normal reply.

    A hint

    The video said to set the timeout from a measured normal reply.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Four requests per case, one seed and one small local model (llama3.2). The busy and down servers are faults injected by the code; only the short timeout is a real httpx timeout. The backoff has no jitter, the waits are far shorter than production values, and seconds were not measured.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos