learning-it by example

#59 LLM engineering and evals 4.6 min

Stream LLM Replies in Python: First Word in 0.2s

The same prompt showed its first word after 0.2 seconds instead of eight.

1. What does streaming a model's reply change, and what does it not change?
2. In the video's timing run (llama3.2 in Ollama on the author's laptop, a three-sentence prompt, model already loaded), roughly when did the first text appear without streaming, and with streaming?
3. In the video's streaming code, how does the stream function stop a reply early?
4. What did the video say it did not measure about stopping a stream?
5. Why does the video's comparison code send a throwaway request before timing anything?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Spot the stream in the code

    Copy the code folder first. Open stream.py and find the one word that turns on streaming in the ollama.generate call, and the loop that reads the pieces. Write down what chunk.response holds, and why first is set only when chunk.response is not empty.

    A hint

    The video said the main change in the code was one word and a loop.

  2. Predict a different limit

    In compare.py the last line stops the stream after 15 chunks. Before running anything, predict what happens to the time and the word count if you change 15 to 5, and what happens if you change it to 40. Write down your two predictions.

    A hint

    In the video, 15 chunks gave 14 words after about 2 seconds.

  3. Time your own prompt

    Add a fourth entry to PROMPTS in compare.py with a prompt of your own, such as asking for an explanation of a stack in three sentences. Predict whether the streamed first text will arrive before or after the whole reply of the not streamed version. If Ollama and llama3.2 are installed, run python compare.py and compare the two numbers on your machine.

    A hint

    Your seconds will differ from the video, because they depend on the machine.

  4. Print the pieces as they arrive

    In stream.py add print(chunk.response, end='', flush=True) inside the loop in streamed, before the line that adds to text. Predict how the reply will look on screen compared with printing it at the end. If you can run Ollama, run python compare.py and watch it. Then remove flush=True and try again. The pieces are printed without a newline, so the timing lines will be glued to the end of the printed text. Remove the print line again when you have finished.

    A hint

    The video listed printing without flushing as a mistake, because the text can still appear in bursts.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Three short prompts, one small local model (llama3.2), one run each on a busy laptop with the model already loaded, so the seconds are rough. No web server or browser page was built, and whether the server stops generating after the client closes the stream was not measured.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos