learning-it by example

Back to the video

Structured Outputs in Python: Valid JSON Is Not Correct JSON

Turn messy emails into structured tickets with a local model and Pydantic, then test the answers. Valid JSON is not the same as correct JSON.

Video #05 ยท LLM engineering and evals

Steps in the video

  1. 0:00 Valid is not correct
  2. 0:17 Why force a structure
  3. 0:46 Setup
  4. 0:58 The schema
  5. 1:43 Without a schema
  6. 2:15 The test cases
  7. 3:12 What the tests caught
  8. 3:47 What this does not show
  9. 4:22 Mistakes to avoid
  10. 4:50 My take and next steps
  11. 5:13 Recap

Files

  • emails.py (11 lines)
  • run_tests.py (13 lines)
  • tickets.py (30 lines)
  • without_schema.py (12 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1

What this does not show

Seven made-up emails with labels chosen by the author, one small local model (llama3.2), one run at temperature 0. The 5 of 7 score says nothing about your data or another model. A schema constrains the shape, not the truth of the answer.

Sources

Learning IT by Example.