learning-it by example

#235 Enterprise architecture and AI agents 5.6 min

An End-to-End Audit Trail for AI Agent Actions in Python

One line in my log had no request id, and it was the audit record of a refund.

1. In a Python program, how can one request id reach every log line from the gateway, agent, model and tool without being passed to each function?
2. In the Python request-id logging demo, three requests ran at once with the worker thread as written. How many log lines were there and how many had no request id?
3. In the made-up gateway, agent, model and tool platform, which part lost the request id, and what put it back?
4. In the Python request-id logging demo, what was and was not tested, and what is needed between separate services?
5. Which cheap test finds gaps in a log's request-id trail before an incident does?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Change the refund wording

    Copy the code folder first. In compare.py, change the second request's prompt from "refund my order" to "please refund my order". Predict the last line the program prints, then run python compare.py and check. When I ran it, the last line was: who and what caused the refund: ben asked: please refund my order.

    A hint

    The gateway logs the user and the prompt, and the answer line reads the gateway line of request r2.

  2. Add a fourth request

    In compare.py, add a fourth request to the REQUESTS list, for example ("r4", "dan", "refund again"). Predict how many lines there will be in each run and how many will have no request id when the worker thread is as written. When I ran it I got 24 lines, 2 without a request id for the first run and 0 for the second. Then remove the fourth request again before the next task.

    A hint

    Each request with the word refund in the prompt goes through the audit sink worker thread.

  3. Find the line that sets the id

    Open platform_parts.py and find the one line that sets the request id. Then open trail.py and find the line in the log handler that reads it. Write one sentence saying why the gateway, agent, model and tool never mention the id.

    A hint

    Look for REQUEST.set in platform_parts.py and REQUEST.get in trail.py.

  4. Break the fix on purpose

    In platform_parts.py, predict what happens if you remove the copy_context().run part from the fixed branch and call loop.run_in_executor(None, audit_sink, "A-1") there as well. Run python compare.py and check whether the second run still reports 0 lines without a request id. When I ran it, the second run reported 1 line without a request id, although its printed label still says the context is copied, because that text is fixed.

    A hint

    The fix is the only difference between the two runs.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One made-up program in one process, no model. Only a run_in_executor thread was tested; asyncio.to_thread copies the context by itself. Queues, scheduled jobs and separate services, which need a header such as traceparent, were not tested. The trail is not tamper-evident.

Up next in Enterprise architecture and AI agents

More in Enterprise architecture and AI agents

All Enterprise architecture and AI agents videos

Related videos