learning-it by example

#58 LLM engineering and evals 5.2 min

JSON Mode vs Schema vs Function Calling: Measured (Ollama)

JSON mode with no field names in the prompt fit my fields 0 times out of 12. With the keys named: 12 of 12.

1. What does JSON mode promise when you call a local model through Ollama with format set to json?
2. In the video's test of llama3.2 through Ollama on twelve made-up order emails, how did plain JSON mode score when the prompt did not name the keys?
3. In the video's test of llama3.2 on twelve made-up order emails, what kind of misses did the function call make, even though its shape fitted every time?
4. Who runs the function when a model returns a function call, and what does the model give back?
5. In the video's structured output code, what does the Pydantic Order class do?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Predict which email is the trap

    Copy the code folder first. Read the twelve emails in orders.py. Find the one trap email and explain in one sentence what makes it hard, and what quantity and total the expected values say. Then predict which of the four methods will fail on it.

    A hint

    The video said the trap mentions two helmets sent back and one light bought.

  2. Read the scoring rules

    Open compare.py and read is_right. Say in your own words which three things must all be true for a result to count as right. Then decide whether an answer with the item written as 'Red Helmet' for the word 'helmet' would count, and why.

    A hint

    Look at the .lower() call and the in test on the item word.

  3. Add your own order email

    Add a thirteenth tuple to CASES in orders.py: an order email you write, with an item word, quantity and total. Before running, predict whether JSON mode with the keys named will get it right. If Ollama, llama3.2, ollama and pydantic are installed, run python compare.py and check that each method now scores out of 13.

    A hint

    Write the numbers as digits first. Later, try one written as a word, like the 'five spare chains' email.

  4. Name the keys differently

    In methods.py change the keys in json_mode_keys from item, quantity and total to name, count and price. Predict what happens to the 'fits schema' number for that method and why. If you can run Ollama, run python compare.py and see. Then change the keys back to item, quantity and total.

    A hint

    The Order class still expects item, quantity and total.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Twelve made up emails, one small model (llama3.2), one run at temperature 0, so a difference of one or two answers means little. The schema was not also put in the prompt for the schema version, the function call had one tool and no way to decline, and no other model or latency was tested.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos