#16 MLOps and deployment 6.3 min
Deploy an LLM API with FastAPI and Docker: The 0.0.0.0 Trap
Put a local LLM behind a FastAPI endpoint, pack it in a Docker container, and reproduce the bind-address mistake that makes it unreachable.
Call an API that is not running
Copy the code folder first. With no container running, run python call_api.py 18000 and read the three lines. Predict first whether any of them will show a status number, and what the text after unreachable will be. Compare it with the video's broken container, where the connection was dropped. This step needs only Python.
A hint
The script catches OSError and prints the name of the error type in brackets.
Loosen the validation
In app.py change min_length=20 to min_length=5. The text 'too short' has 9 characters. Predict the status of the 'bad input' call compared with the 422 string_too_short in the video. To test it, build a new image tag such as yt-api:2, start it on port 18002 and run python call_api.py 18002. This needs Docker and Ollama, as in the video. Remove the container afterwards with docker rm -f.
A hint
Validation by the request model happens before the model is called.
Show the summary
In call_api.py print the summary text itself as well as the field names. Then in app.py change num_predict from 40 to 10, rebuild, and run it. Predict whether the summary will be longer or shorter than before. The video caps the answer so that a slow model cannot ramble.
A hint
Use a new image tag and a new port, so your earlier container is left alone.
Listen on the wrong address
First build the image: docker build -q -t yt-api:1 . Then start a second container from it with the command overridden to run Uvicorn with --host 127.0.0.1, published on port 18001, as in the video. Predict what each of the three calls in call_api.py prints, then run python call_api.py 18001. Remove the container with docker rm -f afterwards.
A hint
Inside a container, 127.0.0.1 means only the container itself can connect.
These tasks are for you to try on your own machine. Nothing is checked or saved.
What this does not show
One container on one laptop calling Ollama on the same machine; no cloud deployment, sign-in, load test or scaling. host.docker.internal works out of the box with Docker Desktop; plain Docker on Linux needs an extra setting. One mistake (the bind address) is reproduced; the summary text itself is not shown or checked.
Up next in MLOps and deployment
More in MLOps and deployment
Related videos
- #157 Model Registry by Hand: Versions, Promote, Roll Back MLOps and deployment
- #158 Detect Data Drift: PSI and K-S by Hand in Python MLOps and deployment
- #159 Feature Stores: When a Table and a Join Are Enough MLOps and deployment
- #163 Secrets in ML Apps: Scan, Move Out, Rotate (Python) MLOps and deployment