learning-it by example

#230 Enterprise architecture and AI agents 5.6 min

Agent Gateway: One Front Door for Models, Tools, Agents

I sent eight requests through one gateway, and five were stopped at the door.

1. What does an agent gateway do that the video compares to gateway offloading?
2. Of the eight requests sent through the Python FastAPI agent gateway in the demo, how many passed on to a service and how many were stopped?
3. The Python FastAPI agent gateway checks each request in a fixed order. Which order of failures does it use?
4. Why does the Python FastAPI agent gateway write the audit line before it calls the service behind it?
5. The video says its toy gateway is not enough for a real system. What would a real gateway also need?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Allow a longer prompt

    Copy the code folder first. This code needs fastapi, starlette and httpx at the versions in requirements.txt. In gateway.py, change MAX_PROMPT from 200 to 600. Predict the status of the request with the 500-character prompt and the last line it prints, then run python compare.py and compare. The model service answers any request that reaches it with a status 200.

    A hint

    The 413 check compares the length of the prompt with MAX_PROMPT.

  2. Give the refund bot a model scope

    In gateway.py, add "llm:chat" to the scopes of bot-key. Predict what happens to the bot-key request to /llm/chat, and how many requests now show as passed on. Run python compare.py to check your prediction.

    A hint

    The scope check compares the scope the destination needs with the list of scopes for the caller.

  3. Add a ninth request

    In compare.py, add a ninth request to REQUESTS: the key stolen-key sent to /payments/send with the body {"amount": 5}. Predict which status the gateway returns, even though that destination does not exist. Think about which of the four checks runs first. Then predict how many lines the audit log has. Run python compare.py to check.

    A hint

    The destination is only checked after the key has been looked up.

  4. Change who may call agents

    In compare.py, add a request that sends bot-key to /agents/ask with the body {"prompt": "hi"}. Predict the status and the message. Then in gateway.py give bot-key the scope "agents:call" and predict again. Run python compare.py after each change and write down what the audit log count says each time.

    A hint

    The scope each destination needs is stored next to its service in UPSTREAMS.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Two made-up callers with keys in a dictionary, and services called in the same process. Only the prompt field is size-checked, a body that is not a JSON object fails with no audit line, and the audit records the gateway's verdict, not the reply.

Up next in Enterprise architecture and AI agents

More in Enterprise architecture and AI agents

All Enterprise architecture and AI agents videos

Related videos