#64 LLM engineering and evals 5.1 min
Chain of Thought: When Step by Step Helps, and Its Cost
On five multi-step problems my model got 0 right answering directly and 4 right thinking step by step.
Work out the answers yourself
Copy the code folder first. Open problems.py and solve the five several step problems on paper. Then compare with the answers that Python works out from the same numbers in the file. Python gives 5, 11, 111, 385 and 55, in that order. Check whether any of your answers differs and why.
A hint
Read each calculation in the file's third field to see how the answer is built.
Test how the number is read
Make a small script that imports number_in from ask.py and prints number_in('Step 1: 5 x 3 = 15\nAnswer: 5', 'steps first'), then the same text without the last line, and number_in('Answer: 1,250 pounds', 'steps first'). Predict the three numbers before you run it. You should see 5.0, then 1.0, then 1250.0. Explain why the second one is the first number in the text.
A hint
If the word Answer is missing, the code falls back to the first number in the text, which is probably not the final answer.
Add a problem of your own
Add a sixth several step problem to PROBLEMS in problems.py, with the answer worked out by Python from the same numbers in the question. Predict whether answer only will get it right. If Ollama and llama3.2 are installed, run python compare.py and check that the several steps rows now say out of 6.
A hint
Write the answer as a calculation, not a typed number, as the file does.
Write a new style
Add a fourth style to STYLES and LIMIT in ask.py, for example 'short steps': 'Use at most two short lines of working, then write the last line as: Answer: <number>' with a limit of 100. Predict whether it will beat answer only on the several step problems and use fewer tokens than steps first. If you can run Ollama, run python compare.py and compare.
A hint
A limit that is too small cuts off the reasoning before the word Answer.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Ten made-up word problems that look alike, five per row, so one answer is a fifth of a row. One small local model (llama3.2), one run, one wording per style. Seconds are from a shared laptop and will differ. No larger model, reasoning model or worked examples were tested. A correct final number does not prove the steps were right.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
- #57 LLM as Judge: I Swapped the Order and It Changed LLM engineering and evals
- #68 Synthetic Test Cases: 27 Asked, 17 Passed Checks LLM engineering and evals
- #69 Detect LLM Hallucinations: 5 Checks Scored on 16 Answers LLM engineering and evals