#59 LLM engineering and evals 4.6 min
Stream LLM Replies in Python: First Word in 0.2s
The same prompt showed its first word after 0.2 seconds instead of eight.
Spot the stream in the code
Copy the code folder first. Open stream.py and find the one word that turns on streaming in the ollama.generate call, and the loop that reads the pieces. Write down what chunk.response holds, and why first is set only when chunk.response is not empty.
A hint
The video said the main change in the code was one word and a loop.
Predict a different limit
In compare.py the last line stops the stream after 15 chunks. Before running anything, predict what happens to the time and the word count if you change 15 to 5, and what happens if you change it to 40. Write down your two predictions.
A hint
In the video, 15 chunks gave 14 words after about 2 seconds.
Time your own prompt
Add a fourth entry to PROMPTS in compare.py with a prompt of your own, such as asking for an explanation of a stack in three sentences. Predict whether the streamed first text will arrive before or after the whole reply of the not streamed version. If Ollama and llama3.2 are installed, run python compare.py and compare the two numbers on your machine.
A hint
Your seconds will differ from the video, because they depend on the machine.
Print the pieces as they arrive
In stream.py add print(chunk.response, end='', flush=True) inside the loop in streamed, before the line that adds to text. Predict how the reply will look on screen compared with printing it at the end. If you can run Ollama, run python compare.py and watch it. Then remove flush=True and try again. The pieces are printed without a newline, so the timing lines will be glued to the end of the printed text. Remove the print line again when you have finished.
A hint
The video listed printing without flushing as a mistake, because the text can still appear in bursts.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
Three short prompts, one small local model (llama3.2), one run each on a busy laptop with the model already loaded, so the seconds are rough. No web server or browser page was built, and whether the server stops generating after the client closes the stream was not measured.
Up next in LLM engineering and evals
More in LLM engineering and evals
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON
- #06 Test Your LLM App with pytest: An Eval Harness in 40 Lines
- #17 Cut LLM Cost and Latency: Caching and Shorter Answers
- #56 Prompt Versioning: Compare Changes Case by Case
- #57 LLM as Judge: I Swapped the Order and It Changed
- #58 JSON Mode vs Schema vs Function Calling: Measured (Ollama)
