learning-it by example

#61 LLM engineering and evals 5.3 min

Count Tokens Before You Send: Context Limits

I sent 1,086 tokens into a 1,024-token context on purpose. No error, and the first rule was not followed.

1. How can you get the real number of tokens in a prompt for a local Ollama model before using it?
2. In the video's token count (llama3.2 in Ollama, a made-up text of order ids, dates and hashes repeated twelve times), how did the word based guess compare with the real token count?
3. In the video's Ollama test with llama3.2 and the context set to 1024 tokens, what happened when a prompt of 80 short notes (1,086 tokens) was sent?
4. In the video's Ollama test with llama3.2, why can the video not prove that the cut, rather than the length, lost the rule for the 80 notes?
5. What advice did the video give about where to put the most important rule in a long prompt?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Check the word guess by hand

    Copy the code folder first. Open samples.py and count the words in ENGLISH, CODE and IDS by splitting on spaces and line breaks. Then work out what guess() in count.py would give for each one repeated 12 times (words times 1.3, rounded). Compare with the video: 168, 120 and 48 words gave guesses of 218, 156 and 62.

    A hint

    ENGLISH has 14 words, CODE has 10 and IDS has 4 when split on whitespace.

  2. Halve the repeats

    In samples.py change each * 12 to * 6. Predict the new word counts and guesses, then run python compare.py if the Ollama app and llama3.2 are installed. Write your prediction first, then check it: the word counts should be 84, 60 and 24, and the guesses 109, 78 and 31. Predict whether the real counts for the ids text will still be many times higher than its guess.

    A hint

    The guess only depends on the words, so it halves. The real count depends on how the model's tokenizer splits the text.

  3. Write a fourth kind of text

    Add a fourth entry to SAMPLES in samples.py, such as a line of text in another language or a line of emoji-free punctuation like '!!! ??? ... ---'. Before you run it, predict whether the guess will be above or below the real count. Run python compare.py and write down the three numbers for your new text.

    A hint

    The video's tip was to count with the model you are really using.

  4. Move the rule to the end

    In compare.py change the prompt line so the rule is repeated after the notes, just before the question, as well as at the start. Predict whether the 80 notes case will now follow the rule. If you can run Ollama, run python compare.py and see. Remember that this is one run and that it cannot prove the reason.

    A hint

    Put the same RULE text again before 'Question:' in the prompt.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Three made up texts, one model (llama3.2), one run. Counts are Ollama's prompt_eval_count, which includes the model's prompt template, and the counter has its own 4096-token context. The 80 notes were not re-run in a larger context, so the lost rule is inferred from 1024 of 1086 tokens being seen.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos