learning-it by example

#11 Local and open models 6 min

Run an LLM Locally with Ollama and Python: First Call

Run a language model on your own laptop and call it from Python: a first answer, streaming, the OpenAI client with one line changed, and two common errors reproduced.

1. How does Python code talk to a model that Ollama is running?
2. On a laptop with no graphics card, the first call to llama3.2 wrote 39 tokens in 11.1 seconds. Roughly what speed is that?
3. What two things change when you point the OpenAI Python client at Ollama?
4. What does the ollama client report for a model that was never pulled, and what is the fix?
5. What does stream=True change about ollama.chat?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Run the same call again

    Copy the code folder first. Run python basic.py three times and write down the tokens and the seconds each time. Then change temperature to 1 in the options, remove the seed entry as well, and run it three more times. Predict which set will repeat more. This needs Ollama with llama3.2, as in the video.

    A hint

    The video says temperature zero and a fixed seed mean the same question usually gives the same answer, though that is not guaranteed.

  2. Forget the /v1

    In compat.py remove /v1 from the end of base_url. Predict whether the call will work, then run python compat.py and note the error type. Put /v1 back afterwards.

    A hint

    The video says the base URL must end in slash v one.

  3. See the pieces

    In stream.py print each piece inside square brackets, for example print('[' + part.message.content + ']', end='', flush=True). Run it and look at how the answer is split up. Is every piece a whole word?

    A hint

    The video says each piece holds a few characters.

  4. Time the first piece

    In stream.py import time, note the time before the loop, and print how many seconds passed when the first piece arrives, and again at the end. Predict which of the two is bigger and what that tells you about why streaming helps.

    A hint

    Use time.perf_counter(). The video says streaming does not make the model faster, but makes the wait feel shorter.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One small model (llama3.2) on one laptop with no graphics card: about 3.5 tokens a second here, and your speed will differ. Only one simple chat call was tested through the OpenAI client; Ollama supports a subset of that API.

Up next in Local and open models

More in Local and open models

All Local and open models videos

Related videos