learning-it by example

#26 AI agents and MCP 5.8 min

Plan, Act, Reflect: Does It Help a Small Model?

One example in the prompt took a small model from 2 of 6 to 6 of 6. Reflection added nothing.

1. On six made up shopping questions, which change took a small model's plan score from 2 of 6 to 6 of 6?
2. When the model was asked to reflect on its own example plans, what did it do?
3. In the planning agent, why can a plan from the model never run arbitrary code?
4. A bare plan was checked in code and the model was told what was wrong, up to three times. How many of the four bad plans did it repair?
5. What does the video say it did not test, so its results might differ for them?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Use the safe calculator

    Copy the code folder first. In a Python prompt import calc and price from tools. Work out 3 helmets and 2 locks by calling calc('3*helmet + 2*lock', {'helmet': price('helmet'), 'lock': price('lock')}). Check you get 160.0, the answer for the first question in tasks.py.

    A hint

    The second argument is a dictionary from item names to prices.

  2. Try to break the calculator

    Call calc with the expressions '2**3' and 'abs(3)', and one that uses an item name you did not give in the names dictionary. Predict first what happens to each. When we ran them, all three raised a ValueError that began with not allowed. Write a sentence on why that matters when a model writes the expression.

    A hint

    Only numbers, the given names and plus, minus, times and divide are allowed. Power and function calls are not.

  3. Plan a new question by hand

    Write the plan yourself for the question: what do 2 gloves and 1 pump cost? Use a lookups list and one expression, then run it with calc and price from tools.py. We got 50.98. Then do the same for the average price of a helmet, a lock and a light, and compare your result with the 29.16 in tasks.py. We got 29.16333333333333, which passes the check in compare.py because it is within 0.01 of the answer.

    A hint

    For the average, add the three names and divide by 3.

  4. Predict the code check

    The code check problem_with in agent.py looks at known names and unused lookups. Predict what it returns for a plan with lookups ['bottle'] and expression '2*bottles', for lookups ['bottle', 'tube'] and expression '2*bottle', and for lookups ['bottle'] and expression '2*bottle'. Then check by importing Plan and problem_with from agent, which needs the packages from the video. We expected unknown names bottles, then looked up but not used: tube, then None.

    A hint

    The check collects the lower case words in the expression and compares them with the price list and the lookups.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six made up shopping questions, one worked example that looks like them, one small model (llama3.2), one run at temperature zero. No larger model and no tool-choosing loop was tested.

Up next in AI agents and MCP

More in AI agents and MCP

All AI agents and MCP videos

Related videos