learning-it by example

Back to the video

Deploy an LLM API with FastAPI and Docker: The 0.0.0.0 Trap

Put a local LLM behind a FastAPI endpoint, pack it in a Docker container, and reproduce the bind-address mistake that makes it unreachable.

Video #16 ยท MLOps and deployment

Steps in the video

  1. 0:00 An LLM behind an API
  2. 0:24 What a container buys you
  3. 0:42 Setup
  4. 1:07 The app: setup and validation
  5. 1:47 The app: two endpoints
  6. 2:06 The Dockerfile
  7. 2:54 Call it like a client
  8. 3:37 Break it: the wrong address
  9. 4:31 What this does not show
  10. 4:58 Mistakes to avoid
  11. 5:34 My take and next steps
  12. 5:55 Recap

Files

  • Dockerfile (7 lines)
  • app.py (30 lines)
  • call_api.py (28 lines)
  • requirements.txt (4 lines)

Download the code (zip)

Tested with

Python 3.12.15 | fastapi==0.162.0, uvicorn==0.54.0, ollama==0.6.3, pydantic==2.14.1

What this does not show

One container on one laptop calling Ollama on the same machine; no cloud deployment, sign-in, load test or scaling. host.docker.internal works out of the box with Docker Desktop; plain Docker on Linux needs an extra setting. One mistake (the bind address) is reproduced; the summary text itself is not shown or checked.

Sources

Learning IT by Example.