Chain of Thought: When Step by Step Helps, and Its Cost
On five multi-step problems my model got 0 right answering directly and 4 right thinking step by step.
Video #64 ยท LLM engineering and evals
Steps in the video
Files
- ask.py (26 lines)
- compare.py (12 lines)
- problems.py (24 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Ten made-up word problems that look alike, five per row, so one answer is a fifth of a row. One small local model (llama3.2), one run, one wording per style. Seconds are from a shared laptop and will differ. No larger model, reasoning model or worked examples were tested. A correct final number does not prove the steps were right.
Sources
Learning IT by Example.