learning-it by example

#64 LLM engineering and evals 5.1 min

Chain of Thought: When Step by Step Helps, and Its Cost

On five multi-step problems my model got 0 right answering directly and 4 right thinking step by step.

1. What does chain of thought mean when you prompt a model?
2. In the video's test of llama3.2 on five made-up word problems that need several steps, how many did it get right with answer only, with steps first, and with answer then explain?
3. In the video's test of llama3.2 on five made-up one step word problems, such as adding two numbers, what did steps first change compared with answer only?
4. Why is asking for the answer first and the reasoning after a trap?
5. How does the code in the video decide whether a model's answer is right?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Work out the answers yourself

    Copy the code folder first. Open problems.py and solve the five several step problems on paper. Then compare with the answers that Python works out from the same numbers in the file. Python gives 5, 11, 111, 385 and 55, in that order. Check whether any of your answers differs and why.

    A hint

    Read each calculation in the file's third field to see how the answer is built.

  2. Test how the number is read

    Make a small script that imports number_in from ask.py and prints number_in('Step 1: 5 x 3 = 15\nAnswer: 5', 'steps first'), then the same text without the last line, and number_in('Answer: 1,250 pounds', 'steps first'). Predict the three numbers before you run it. You should see 5.0, then 1.0, then 1250.0. Explain why the second one is the first number in the text.

    A hint

    If the word Answer is missing, the code falls back to the first number in the text, which is probably not the final answer.

  3. Add a problem of your own

    Add a sixth several step problem to PROBLEMS in problems.py, with the answer worked out by Python from the same numbers in the question. Predict whether answer only will get it right. If Ollama and llama3.2 are installed, run python compare.py and check that the several steps rows now say out of 6.

    A hint

    Write the answer as a calculation, not a typed number, as the file does.

  4. Write a new style

    Add a fourth style to STYLES and LIMIT in ask.py, for example 'short steps': 'Use at most two short lines of working, then write the last line as: Answer: <number>' with a limit of 100. Predict whether it will beat answer only on the several step problems and use fewer tokens than steps first. If you can run Ollama, run python compare.py and compare.

    A hint

    A limit that is too small cuts off the reasoning before the word Answer.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Ten made-up word problems that look alike, five per row, so one answer is a fifth of a row. One small local model (llama3.2), one run, one wording per style. Seconds are from a shared laptop and will differ. No larger model, reasoning model or worked examples were tested. A correct final number does not prove the steps were right.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos