learning-it by example

#245 Enterprise architecture and AI agents 5.8 min

AI Agent Observability: OpenTelemetry Traces, Cost, Replay

When an agent gives a bad answer, the hard part is getting that exact run back.

1. In OpenTelemetry tracing of an AI agent, what is a trace and what does each span in it stand for?
2. For the one traced refund request in the Python OpenTelemetry agent demo, how many spans were there, and how many tokens did the model calls use?
3. Why does putting the content on the spans, and not only the cost, make a replay possible?
4. In the OpenTelemetry agent demo, what happened when the traced refund request was replayed from the recorded spans?
5. What is the risk of putting the full content of every prompt in a trace?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Find the six spans

    Copy the code folder first. Open agent.py and find the place where the root span agent.run starts, the place where each model.call span starts, and the place where each tool.call span starts. Using the SCRIPT dictionary, work out why one request makes three model.call spans, two tool.call spans and one root span, which is six spans in all. This is a reading task.

    A hint

    The first two steps ask for a tool, and the third step gives the answer.

  2. Predict a changed cost

    In agent.py, change the token number in SCRIPT step 1 from 520 to 1000. Predict the total tokens and the total credits for model.call, using the price CREDITS_PER_1K = 3.0 in the file. Then, if you have the two OpenTelemetry packages from the video installed, run python compare.py and compare. I did not run this change, so treat the numbers as your own prediction.

    A hint

    Credits are tokens divided by 1000 and multiplied by 3.0.

  3. Predict the replay with a different question

    In compare.py, change the question "Please refund my order A-1" to another sentence. Predict whether the made-up model's answer changes and whether the replay still gives the same answer. Explain your reasoning from the SCRIPT dictionary. If you have the packages installed, run python compare.py to check. I did not run this change.

    A hint

    Look at how the made-up model chooses what to say at each step.

  4. Say what the replay leaves out

    Read replay.py and the paragraph in the video about what a replay cannot do. Write two sentences saying why this replay returns the same answer by design, and what kinds of real-life differences, such as time or randomness, a replay built only from recorded outputs would not reproduce. This is a reading task.

    A hint

    The replay feeds recorded outputs to the agent instead of asking the model.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One request with a made-up model and tools, traced in memory; the replay returns the recorded outputs, so the same answer and zero real calls are by design; it tests agent code, not the model.

Up next in Enterprise architecture and AI agents

More in Enterprise architecture and AI agents

All Enterprise architecture and AI agents videos

Related videos