learning-it by example

Back to the video

Count Tokens Before You Send: Context Limits

I sent 1,086 tokens into a 1,024-token context on purpose. No error, and the first rule was not followed.

Video #61 ยท LLM engineering and evals

Steps in the video

  1. 0:00 A prompt too long, and no error
  2. 0:26 The idea
  3. 1:00 Setup
  4. 1:18 Count first
  5. 1:42 Three kinds of text
  6. 2:02 Send too much
  7. 3:12 What this does not show
  8. 3:59 Mistakes to avoid
  9. 4:22 My take and next steps
  10. 4:56 Recap

Files

  • compare.py (21 lines)
  • count.py (14 lines)
  • samples.py (5 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Three made up texts, one model (llama3.2), one run. Counts are Ollama's prompt_eval_count, which includes the model's prompt template, and the counter has its own 4096-token context. The 80 notes were not re-run in a larger context, so the lost rule is inferred from 1024 of 1086 tokens being seen.

Sources

Learning IT by Example.