#58 LLM engineering and evals 5.2 min
JSON Mode vs Schema vs Function Calling: Measured (Ollama)
JSON mode with no field names in the prompt fit my fields 0 times out of 12. With the keys named: 12 of 12.
Predict which email is the trap
Copy the code folder first. Read the twelve emails in orders.py. Find the one trap email and explain in one sentence what makes it hard, and what quantity and total the expected values say. Then predict which of the four methods will fail on it.
A hint
The video said the trap mentions two helmets sent back and one light bought.
Read the scoring rules
Open compare.py and read is_right. Say in your own words which three things must all be true for a result to count as right. Then decide whether an answer with the item written as 'Red Helmet' for the word 'helmet' would count, and why.
A hint
Look at the .lower() call and the in test on the item word.
Add your own order email
Add a thirteenth tuple to CASES in orders.py: an order email you write, with an item word, quantity and total. Before running, predict whether JSON mode with the keys named will get it right. If Ollama, llama3.2, ollama and pydantic are installed, run python compare.py and check that each method now scores out of 13.
A hint
Write the numbers as digits first. Later, try one written as a word, like the 'five spare chains' email.
Name the keys differently
In methods.py change the keys in json_mode_keys from item, quantity and total to name, count and price. Predict what happens to the 'fits schema' number for that method and why. If you can run Ollama, run python compare.py and see. Then change the keys back to item, quantity and total.
A hint
The Order class still expects item, quantity and total.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Twelve made up emails, one small model (llama3.2), one run at temperature 0, so a difference of one or two answers means little. The schema was not also put in the prompt for the schema version, the function call had one tool and no way to decline, and no other model or latency was tested.
