#61 LLM engineering and evals 5.3 min
Count Tokens Before You Send: Context Limits
I sent 1,086 tokens into a 1,024-token context on purpose. No error, and the first rule was not followed.
Check the word guess by hand
Copy the code folder first. Open samples.py and count the words in ENGLISH, CODE and IDS by splitting on spaces and line breaks. Then work out what guess() in count.py would give for each one repeated 12 times (words times 1.3, rounded). Compare with the video: 168, 120 and 48 words gave guesses of 218, 156 and 62.
A hint
ENGLISH has 14 words, CODE has 10 and IDS has 4 when split on whitespace.
Halve the repeats
In samples.py change each * 12 to * 6. Predict the new word counts and guesses, then run python compare.py if the Ollama app and llama3.2 are installed. Write your prediction first, then check it: the word counts should be 84, 60 and 24, and the guesses 109, 78 and 31. Predict whether the real counts for the ids text will still be many times higher than its guess.
A hint
The guess only depends on the words, so it halves. The real count depends on how the model's tokenizer splits the text.
Write a fourth kind of text
Add a fourth entry to SAMPLES in samples.py, such as a line of text in another language or a line of emoji-free punctuation like '!!! ??? ... ---'. Before you run it, predict whether the guess will be above or below the real count. Run python compare.py and write down the three numbers for your new text.
A hint
The video's tip was to count with the model you are really using.
Move the rule to the end
In compare.py change the prompt line so the rule is repeated after the notes, just before the question, as well as at the start. Predict whether the 80 notes case will now follow the rule. If you can run Ollama, run python compare.py and see. Remember that this is one run and that it cannot prove the reason.
A hint
Put the same RULE text again before 'Question:' in the prompt.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Three made up texts, one model (llama3.2), one run. Counts are Ollama's prompt_eval_count, which includes the model's prompt template, and the counter has its own 4096-token context. The 80 notes were not re-run in a larger context, so the lost rule is inferred from 1024 of 1086 tokens being seen.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
