learning-it by example

Back to the video

Run an LLM Locally with Ollama and Python: First Call

Run a language model on your own laptop and call it from Python: a first answer, streaming, the OpenAI client with one line changed, and two common errors reproduced.

Video #11 ยท Local and open models

Steps in the video

  1. 0:00 A language model on your laptop
  2. 0:21 What Ollama does
  3. 0:53 Setup
  4. 1:15 A first call
  5. 2:05 Streaming
  6. 2:46 The OpenAI client
  7. 3:31 The two errors
  8. 4:07 What this does not show
  9. 4:38 Mistakes to avoid
  10. 5:07 My take and next steps
  11. 5:31 Recap

Files

  • basic.py (12 lines)
  • compat.py (12 lines)
  • errors.py (15 lines)
  • stream.py (13 lines)

Download the code (zip)

Tested with

Python 3.12.15 | ollama==0.6.3, openai==3.40.0

What this does not show

One small model (llama3.2) on one laptop with no graphics card: about 3.5 tokens a second here, and your speed will differ. Only one simple chat call was tested through the OpenAI client; Ollama supports a subset of that API.

Sources

Learning IT by Example.