Run an LLM Locally with Ollama and Python: First Call
Run a language model on your own laptop and call it from Python: a first answer, streaming, the OpenAI client with one line changed, and two common errors reproduced.
Video #11 ยท Local and open models
Steps in the video
Files
- basic.py (12 lines)
- compat.py (12 lines)
- errors.py (15 lines)
- stream.py (13 lines)
Tested with
Python 3.12.15 | ollama==0.6.3, openai==3.40.0
What this does not show
One small model (llama3.2) on one laptop with no graphics card: about 3.5 tokens a second here, and your speed will differ. Only one simple chat call was tested through the OpenAI client; Ollama supports a subset of that API.
Sources
- https://docs.ollama.com/api/errors
- https://docs.ollama.com/api/introduction
- https://docs.ollama.com/api/openai-compatibility
- https://insiderllm.com/guides/ollama-api-connection-refused-fix/
- https://pypi.org/project/ollama/0.6.3/
- https://pypi.org/project/openai/
- https://theneuralbase.com/ollama/learn/beginner/using-openai-sdk-with-ollama/
Learning IT by Example.