#238 Enterprise architecture and AI agents 5.6 min
Event-Driven AI Agents: Queues, Crashes and Duplicates
After a crash, my toy agent wrote two replies to the same ticket. The agent was fine.
Find the worst moment
Copy the code folder first. Open consumer.py and find the three things that happen for each event: the work, the check for the crash, and the line that saves the position. Write down in which order they run and explain in one sentence why a crash between the work and the saved position makes an event repeat.
A hint
The saved position is the log.ack line.
Take away the poison event
In compare.py, replace the line EVENTS[6] = ("poison", "garbled") with the single word pass, then run python compare.py. Predict the reply count and the dead letter count for each agent first. When I ran it the plain agent showed 13 replies with 1 duplicate and 0 dead letters, and the idempotent agent showed 12 replies with 0 duplicates and 0 dead letters. Then put the EVENTS[6] line back as it was before the next task.
A hint
Every one of the 12 events can now be read.
Crash later in the log
In compare.py, change crash_at=4 to crash_at=9 and predict whether the counts change, then run python compare.py. When I ran it the numbers were the same as before: 12 replies with 1 duplicate for the plain agent and 11 replies with 0 duplicates for the idempotent agent.
A hint
Only the one event that was worked on before the crash is repeated, and event 9 is an ordinary ticket. A crash on the poison event 7 would instead repeat its dead letter.
Change the retry limit
In consumer.py, change attempts=3 to attempts=1 in the definition of run. Predict whether the counts change, then run python compare.py. When I ran it the counts were the same as before, because the poison event always fails and ends in the dead letters whatever the limit. Then write one sentence on what a retry limit protects against.
A hint
The poison event fails every time, so more tries never help it.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One consumer and one faked crash in one process; Redis was not run and consumer groups were not built; the duplicate check works only because the reply is stored next to the log; SQLite is for one machine.
Up next in Enterprise architecture and AI agents
More in Enterprise architecture and AI agents
- #229 A Reference Architecture for Enterprise AI Agents, as Code
- #230 Agent Gateway: One Front Door for Models, Tools, Agents
- #231 MCP Gateway: Per-Tool Policy for Agent Tools (Python)
- #232 Give Every AI Agent Its Own Identity
- #233 Delegated Authority: Let an AI Agent Act for a User, Safely
- #234 Authorisation for AI Agents With Policy as Code (Cedar)
Related videos
- #236 Agent Orchestration Patterns: Supervisor, Pipeline, Swarm Enterprise architecture and AI agents
- #237 Where Should AI Agent State and Memory Live? A Crash Test Enterprise architecture and AI agents
- #239 Async AI Agents: Design for Timeouts With Job Polling Enterprise architecture and AI agents
- #243 Reliability Patterns for AI Agents: Bulkhead, Retry, Cache Enterprise architecture and AI agents
