learning-it by example

Back to the video

JSON Mode vs Schema vs Function Calling: Measured (Ollama)

JSON mode with no field names in the prompt fit my fields 0 times out of 12. With the keys named: 12 of 12.

Video #58 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Three ways
  2. 0:27 The idea
  3. 0:58 Setup
  4. 1:23 The twelve emails
  5. 1:43 The target shape
  6. 1:56 Four versions
  7. 2:43 What the misses were
  8. 3:14 What this does not show
  9. 3:45 Mistakes to avoid
  10. 4:08 My take and next steps
  11. 4:44 Recap

Files

  • compare.py (35 lines)
  • methods.py (44 lines)
  • orders.py (16 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1

What this does not show

Twelve made up emails, one small model (llama3.2), one run at temperature 0, so a difference of one or two answers means little. The schema was not also put in the prompt for the schema version, the function call had one tool and no way to decline, and no other model or latency was tested.

Sources

Learning IT by Example.