#11 Local and open models 6 min
Run an LLM Locally with Ollama and Python: First Call
Run a language model on your own laptop and call it from Python: a first answer, streaming, the OpenAI client with one line changed, and two common errors reproduced.
Run the same call again
Copy the code folder first. Run python basic.py three times and write down the tokens and the seconds each time. Then change temperature to 1 in the options, remove the seed entry as well, and run it three more times. Predict which set will repeat more. This needs Ollama with llama3.2, as in the video.
A hint
The video says temperature zero and a fixed seed mean the same question usually gives the same answer, though that is not guaranteed.
Forget the /v1
In compat.py remove /v1 from the end of base_url. Predict whether the call will work, then run python compat.py and note the error type. Put /v1 back afterwards.
A hint
The video says the base URL must end in slash v one.
See the pieces
In stream.py print each piece inside square brackets, for example print('[' + part.message.content + ']', end='', flush=True). Run it and look at how the answer is split up. Is every piece a whole word?
A hint
The video says each piece holds a few characters.
Time the first piece
In stream.py import time, note the time before the loop, and print how many seconds passed when the first piece arrives, and again at the end. Predict which of the two is bigger and what that tells you about why streaming helps.
A hint
Use time.perf_counter(). The video says streaming does not make the model faster, but makes the wait feel shorter.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One small model (llama3.2) on one laptop with no graphics card: about 3.5 tokens a second here, and your speed will differ. Only one simple chat call was tested through the OpenAI client; Ollama supports a subset of that API.
Up next in Local and open models
More in Local and open models
Related videos
- #59 Stream LLM Replies in Python: First Word in 0.2s LLM engineering and evals
- #01 Local AI Agent in Python: No API Key (Ollama) AI agents and MCP
- #05 Structured Outputs in Python: Valid JSON Is Not Correct JSON LLM engineering and evals
- #02 RAG From Scratch in Python: Chunk, Embed, Retrieve (Local) RAG and vector search