learning-it by example

#36 AI agents and MCP 5.3 min

Multi-Agent Debate: Do Two Agents Beat One? A Test

Two agents argued, and did no better than a plain vote, for more calls.

1. How does the multi-agent debate in the video work when the two agents agree after revising?
2. On the six tricky questions with llama3.2, how many right answers and model calls did each way score?
3. Why did the debate cost 26 model calls for 6 questions?
4. How strongly does the video say the result shows that a debate is no better than a vote?
5. In the agents code, what does the number function do with a model reply?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Test the answer reader

    Copy the code folder first. No model is needed for this. In the code folder start Python and run from agents import number. Predict, then print number for these strings: 'ANSWER: 5', 'ANSWER: **5**', 'The answer is 5', 'ANSWER: 5 then ANSWER: 7', and 'ANSWER: 4.5'. I got 5, 5, None, 7 and 4.

    A hint

    The pattern looks for ANSWER: in capitals, then digits, and takes the last match.

  2. Count the calls by hand

    Read the vote and debate functions in agents.py. Without running anything, work out how many model calls a vote uses for one question, and how many a debate uses when the agents agree and when they disagree. Then work out the call total for 6 questions if all six debates ended in disagreement.

    A hint

    The video's own totals were 18 for the vote and 26 for the debate, with two disagreements.

  3. Add a seventh question

    In questions.py add a seventh tuple with a question that has one whole number as its answer, for example 'What is 12 times 12?' with the answer '144'. Predict the new call counts in the three lines of compare.py. Running it needs Ollama with llama3.2 as in the video, so treat the right answer counts as a guess.

    A hint

    The call counts do not depend on whether the model is right, except for the judge call in a debate.

  4. Spot the unfair comparison

    The video says a debate should be compared with the cheapest thing that could work. Read compare.py and name the two baselines it uses, then write two sentences about what you would add to the test before believing a result, using the limits the video states.

    A hint

    The video mentions more questions, a larger model, and a check in code.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six questions, one small model (llama3.2), one fixed set of seeds; a difference of one question proves nothing about debate against a vote. Replies are capped at 90 tokens, and a reply with no final number counts as wrong. Only the call counts are firm.

Up next in AI agents and MCP

More in AI agents and MCP

All AI agents and MCP videos

Related videos