Structured Outputs in Python: Valid JSON Is Not Correct JSON
Turn messy emails into structured tickets with a local model and Pydantic, then test the answers. Valid JSON is not the same as correct JSON.
Video #05 ยท LLM engineering and evals
Steps in the video
Files
- emails.py (11 lines)
- run_tests.py (13 lines)
- tickets.py (30 lines)
- without_schema.py (12 lines)
Tested with
Python 3.12.15 | ollama==0.6.3, pydantic==2.14.1
What this does not show
Seven made-up emails with labels chosen by the author, one small local model (llama3.2), one run at temperature 0. The 5 of 7 score says nothing about your data or another model. A schema constrains the shape, not the truth of the answer.
Sources
- https://docs.narrative.io/reference/architecture/agent-conversations/errors/structured-output-retries-exhausted.md
- https://futureagi.com/glossary/schema-validation-failure/
- https://github.com/ollama/ollama-python/blob/main/examples/structured-outputs.py
- https://pithycyborg.substack.com/p/the-llm-retry-loop-that-looks-like
Learning IT by Example.