learning-it by example

#330 AI agents and MCP 6.8 min

Too Many MCP Tools: Shortlist, Validate and Log (Python)

A plain keyword router put the right tool in a list of 5 for 28 of 30 made-up requests.

1. In the made-up MCP tool-shortlist demo (60 made-up tools), why show an agent a short list of tools instead of all 60?
2. In the made-up MCP tool-shortlist demo, how often was the right tool in the keyword router's list of five for the 30 made-up requests?
3. In the made-up MCP tool-shortlist demo, what does the 28 of 30 actually count?
4. In the made-up MCP tool-call demo, a proposed call to billing.create_invoice sends an amount of 450000, but the declared maximum is 100000. What does the argument check (check_args) do?
5. In the made-up MCP tool-call log, weather.current_weather was called for the request 'Will it rain in Leeds this weekend?', which is labelled weather.forecast, with fine arguments. How was it recorded?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fix two bad calls

    Copy the code folder first. In labelled.py, in CALLS, change "nights": "two" to "nights": 2 and change "amount": 450000 to "amount": 450. Predict how many of the eight calls will pass the check, and what the counts line will say. Run python run.py check and python run.py log.

    A hint

    Each fix removes one problem message. Compare the number of invalid args in the counts line with the original 5.

  2. Send a fractional number of nights

    In labelled.py change "nights" to the value 2.5 in the search_hotels call. Predict the message for that call. Run python run.py check and read the line for travel.search_hotels.

    A hint

    The schema asks for an integer. The message names the field, the type wanted and the value it got.

  3. Show only three tools

    In router.py change k=5 to k=3 in the shortlist function. Predict whether the right tool will be found less often, and what happens to the words in the list. Run python run.py measure. The row labels will still say 5.

    A hint

    A shorter list is smaller, but misses more. I saw 23 of 30 and 27 words for the first list and 24 of 30 for the second.

  4. Ask a question that matches no tool

    In run.py, in the score_demo function, change REQUESTS[8][0] to REQUESTS[18][0], which asks whether it will rain in Leeds. Predict the scores in the list. Run python run.py score. Then explain why the video says not to trust a shortlist when every score is zero.

    A hint

    None of the words in the request appear in any tool description. I saw billing tools with a score of 0.00.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

No model is run, so model accuracy is not measured: the example only checks whether the right tool is in a keyword-router shortlist. Tools, thirty requests and the right-tool labels are made up; the server step made no difference here and a bigger catalogue was not tried. No real MCP server or client is used; the tools are plain Python dicts in the MCP tool shape.

Up next in AI agents and MCP

More in AI agents and MCP

All AI agents and MCP videos

Related videos