Stream LLM Replies in Python: First Word in 0.2s
The same prompt showed its first word after 0.2 seconds instead of eight.
Video #59 ยท LLM engineering and evals
Steps in the video
Files
- compare.py (14 lines)
- stream.py (29 lines)
Tested with
Python 3.12.15 | ollama==0.6.3
What this does not show
Three short prompts, one small local model (llama3.2), one run each on a busy laptop with the model already loaded, so the seconds are rough. No web server or browser page was built, and whether the server stops generating after the client closes the stream was not measured.
Sources
Learning IT by Example.