Count Tokens Before You Send: Context Limits
I sent 1,086 tokens into a 1,024-token context on purpose. No error, and the first rule was not followed.
Video #61 ยท LLM engineering and evals
Steps in the video
Files
- compare.py (21 lines)
- count.py (14 lines)
- samples.py (5 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Three made up texts, one model (llama3.2), one run. Counts are Ollama's prompt_eval_count, which includes the model's prompt template, and the counter has its own 4096-token context. The 80 notes were not re-run in a larger context, so the lost rule is inferred from 1024 of 1086 tokens being seen.
Sources
Learning IT by Example.