learning-it by example

Back to the video

Stream LLM Replies in Python: First Word in 0.2s

The same prompt showed its first word after 0.2 seconds instead of eight.

Video #59 ยท LLM engineering and evals

Steps in the video

  1. 0:00 The first word after 0.2 seconds
  2. 0:29 The idea
  3. 1:08 Setup
  4. 1:25 A streamed reply
  5. 1:52 A whole reply
  6. 2:03 Compare them
  7. 2:59 What this does not show
  8. 3:27 Mistakes to avoid
  9. 3:52 My take and next steps
  10. 4:18 Recap

Files

  • compare.py (14 lines)
  • stream.py (29 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3

What this does not show

Three short prompts, one small local model (llama3.2), one run each on a busy laptop with the model already loaded, so the seconds are rough. No web server or browser page was built, and whether the server stops generating after the client closes the stream was not measured.

Sources

Learning IT by Example.