learning-it by example

#239 Enterprise architecture and AI agents 5.4 min

Async AI Agents: Design for Timeouts With Job Polling

In a simulation, a client that retried on a timeout nearly doubled the agent's work, and got half the answers.

1. What is the asynchronous request reply pattern recommended for slow AI agents?
2. In the simulation of six jobs that need 46 seconds of work, what did the client that waited on one request with a 5 second timeout and 1 retry get?
3. In the simulated slow-agent server, why did the wait-and-retry client make the agent start almost double the work?
4. In the simulated slow-agent server with six jobs, what did submitting a job and polling every 2 seconds cost and achieve?
5. The simulated slow-agent server has no idempotency key and never cancels abandoned work. What would help with each?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Switch off the retry

    Copy the code folder first. In clients.py, change retries=1 to retries=0 in wait_on_one_request. Predict how many answers the waiting client gets and how much agent time it starts, then run python compare.py. When I ran it the first line said 3 of 6 answered, agent time started 46 s. The printed label still says 1 retry, because it is fixed text. Then change retries=0 back to retries=1 before the next task.

    A hint

    Without a retry no timed-out job is started a second time.

  2. Make the timeout longer

    In clients.py, change timeout=5 to timeout=10 in wait_on_one_request. Predict the first line of output, then run python compare.py. The printed label still says 5 s timeout because it is fixed text. When I ran it I got 5 of 6 answered, agent time started 71 s.

    A hint

    Only the 25 second job is longer than 10 seconds. It is started twice.

  3. Poll less often

    In clients.py, change every=2 to every=5 in submit_and_poll. Predict how the number of polls changes and whether all six jobs are still answered, then run python compare.py. When I ran it I got 6 of 6 answered with 46 s of agent time started and 12 polls. The printed label still says every 2 s, because it is fixed text. Then change every=5 back to every=2 before the next task.

    A hint

    The poller gives up after 30 seconds, and the longest job takes 25.

  4. Give up sooner

    In clients.py, change give_up=30 to give_up=10 in submit_and_poll. Predict which job is no longer answered, then run python compare.py. When I ran it I got 5 of 6 answered with 46 s of agent time started and 17 polls. Say why the agent time did not go down.

    A hint

    The server does not know the poller has stopped asking.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

A simulated clock and a pretend server, six jobs I chose; the waiting client gives up after 5 seconds and the poller after 30; the server never cancels abandoned work; no idempotency key; no failures, lost polls or restarts.

Up next in Enterprise architecture and AI agents

More in Enterprise architecture and AI agents

All Enterprise architecture and AI agents videos

Related videos