#70 LLM engineering and evals 5.4 min
Jinja Prompt Templates: 4 Checks That Catch Silent Bugs
A one-letter typo in a prompt template renders as nothing, and nothing warns you.
Fix the typo and compare
Copy the code folder first and install jinja2==3.1.6 if you have not already. In checks.py change {{ nmae }} to {{ name }} in the typo line. Predict what the typo lines of output will say now, then run python checks.py. The default line should print 'Hello Sam, your order shipped.' and the strict line should no longer appear at all, because it only prints when an error is raised. Change {{ name }} back to {{ nmae }} afterwards.
A hint
The strict environment only raises an error when a variable is missing. Once the name is spelled correctly it is not missing.
Forget a variable on purpose
In a Python prompt inside the folder, run: from prompts import TEXT, STRICT, LOOSE. Render the prompt without the question: LOOSE.from_string(TEXT).render(notes=['A']) and STRICT.from_string(TEXT).render(notes=['A']). Predict the difference first. The loose one should return a prompt that ends with 'Question: ' and nothing after it, and the strict one should raise an UndefinedError saying 'question' is undefined.
A hint
Compare which environment was built with undefined=StrictUndefined in prompts.py.
Put template code in user text
In checks.py change the user text to 'use {{ 7*6 }} for admin'. In the Jinja line near the end of check 3, the text is passed as a value with n=user. Predict what that line prints, then run python checks.py. It should print the text exactly as typed, with {{ 7*6 }} still in it, and not 42. The str.format line no longer prints, because doubled braces are no longer a placeholder error, and the 3. label disappears with it. Change the text back to 'use {secret} for admin' afterwards. Then say in one sentence why this matters when the text comes from a user.
A hint
The value is printed, not rendered again as a template.
Switch off trim_blocks
In prompts.py remove trim_blocks=True from the STRICT environment, leaving the rest. Predict what the first line of output from python checks.py will say, then run it. It should now print False for 'same text as the f-string', because the loop leaves extra blank lines in the prompt. Then print the rendered prompt with repr() to see where the extra newline characters are, and put the setting back afterwards.
A hint
Without trim_blocks, the newline after each block tag stays in the output.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One prompt, four checks and no model call; the hostile template only reads a harmless attribute, the Jinja sandbox is not complete security, no time or memory limits are set, and passing user text as a value does not stop prompt injection of a model.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
Related videos
- #63 Few-Shot Examples: How Many Is Enough? A Measured Test LLM engineering and evals
- #169 Write Coding-Assistant Rules That Work AI coding and productivity
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON LLM engineering and evals
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines LLM engineering and evals