JSON Mode vs Schema vs Function Calling: Measured (Ollama)
JSON mode with no field names in the prompt fit my fields 0 times out of 12. With the keys named: 12 of 12.
Video #58 ยท LLM engineering and evals
Steps in the video
Files
- compare.py (35 lines)
- methods.py (44 lines)
- orders.py (16 lines)
Tested with
Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1
What this does not show
Twelve made up emails, one small model (llama3.2), one run at temperature 0, so a difference of one or two answers means little. The schema was not also put in the prompt for the schema version, the function call had one tool and no way to decline, and no other model or latency was tested.
Sources
Learning IT by Example.