learning-it by example

Back to the video

Chain of Thought: When Step by Step Helps, and Its Cost

On five multi-step problems my model got 0 right answering directly and 4 right thinking step by step.

Video #64 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Does thinking step by step help?
  2. 0:28 The idea
  3. 1:04 Setup
  4. 1:24 Ten problems
  5. 1:43 Three ways to ask
  6. 2:02 Reading the answer
  7. 2:28 Score all three
  8. 3:21 What this does not show
  9. 3:57 Mistakes to avoid
  10. 4:18 My take and next steps
  11. 4:45 Recap

Files

  • ask.py (26 lines)
  • compare.py (12 lines)
  • problems.py (24 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Ten made-up word problems that look alike, five per row, so one answer is a fifth of a row. One small local model (llama3.2), one run, one wording per style. Seconds are from a shared laptop and will differ. No larger model, reasoning model or worked examples were tested. A correct final number does not prove the steps were right.

Sources

Learning IT by Example.