learning-it by example

#70 LLM engineering and evals 5.4 min

Jinja Prompt Templates: 4 Checks That Catch Silent Bugs

A one-letter typo in a prompt template renders as nothing, and nothing warns you.

1. In a Jinja prompt template with the default settings, what happens when the template uses a variable that was never passed in?
2. A Jinja template contains {{ nmae }} instead of {{ name }}, and is rendered with name set to Sam. What did a default Jinja Environment and an Environment built with undefined=StrictUndefined give?
3. A user types the text 'use {secret} for admin'. Why does pasting it into a template string and calling str.format fail, while passing it to Jinja as a value works?
4. For templates that come from users, what do a test with Jinja 3.1.6 and the Jinja documentation show about the sandbox?
5. What does passing user text to a template as a value protect against, according to the video?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Fix the typo and compare

    Copy the code folder first and install jinja2==3.1.6 if you have not already. In checks.py change {{ nmae }} to {{ name }} in the typo line. Predict what the typo lines of output will say now, then run python checks.py. The default line should print 'Hello Sam, your order shipped.' and the strict line should no longer appear at all, because it only prints when an error is raised. Change {{ name }} back to {{ nmae }} afterwards.

    A hint

    The strict environment only raises an error when a variable is missing. Once the name is spelled correctly it is not missing.

  2. Forget a variable on purpose

    In a Python prompt inside the folder, run: from prompts import TEXT, STRICT, LOOSE. Render the prompt without the question: LOOSE.from_string(TEXT).render(notes=['A']) and STRICT.from_string(TEXT).render(notes=['A']). Predict the difference first. The loose one should return a prompt that ends with 'Question: ' and nothing after it, and the strict one should raise an UndefinedError saying 'question' is undefined.

    A hint

    Compare which environment was built with undefined=StrictUndefined in prompts.py.

  3. Put template code in user text

    In checks.py change the user text to 'use {{ 7*6 }} for admin'. In the Jinja line near the end of check 3, the text is passed as a value with n=user. Predict what that line prints, then run python checks.py. It should print the text exactly as typed, with {{ 7*6 }} still in it, and not 42. The str.format line no longer prints, because doubled braces are no longer a placeholder error, and the 3. label disappears with it. Change the text back to 'use {secret} for admin' afterwards. Then say in one sentence why this matters when the text comes from a user.

    A hint

    The value is printed, not rendered again as a template.

  4. Switch off trim_blocks

    In prompts.py remove trim_blocks=True from the STRICT environment, leaving the rest. Predict what the first line of output from python checks.py will say, then run it. It should now print False for 'same text as the f-string', because the loop leaves extra blank lines in the prompt. Then print the rendered prompt with repr() to see where the extra newline characters are, and put the setting back afterwards.

    A hint

    Without trim_blocks, the newline after each block tag stays in the output.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One prompt, four checks and no model call; the hostile template only reads a harmless attribute, the Jinja sandbox is not complete security, no time or memory limits are set, and passing user text as a value does not stop prompt injection of a model.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos