learning-it by example

#16 MLOps and deployment 6.3 min

Deploy an LLM API with FastAPI and Docker: The 0.0.0.0 Trap

Put a local LLM behind a FastAPI endpoint, pack it in a Docker container, and reproduce the bind-address mistake that makes it unreachable.

1. In the FastAPI-in-Docker demo, why does the Dockerfile start Uvicorn with the host set to 0.0.0.0?
2. When a client sent text shorter than 20 characters to the FastAPI summary API, what status did the API return?
3. In the FastAPI-in-Docker demo, a second container ran Uvicorn on 127.0.0.1 instead of 0.0.0.0. What did the client see?
4. In the FastAPI-in-Docker video's 'What this does not show' slide, which of these is listed as not shown?
5. According to the video, how should the app in a container reach Ollama running on your own computer, with Docker Desktop?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Call an API that is not running

    Copy the code folder first. With no container running, run python call_api.py 18000 and read the three lines. Predict first whether any of them will show a status number, and what the text after unreachable will be. Compare it with the video's broken container, where the connection was dropped. This step needs only Python.

    A hint

    The script catches OSError and prints the name of the error type in brackets.

  2. Loosen the validation

    In app.py change min_length=20 to min_length=5. The text 'too short' has 9 characters. Predict the status of the 'bad input' call compared with the 422 string_too_short in the video. To test it, build a new image tag such as yt-api:2, start it on port 18002 and run python call_api.py 18002. This needs Docker and Ollama, as in the video. Remove the container afterwards with docker rm -f.

    A hint

    Validation by the request model happens before the model is called.

  3. Show the summary

    In call_api.py print the summary text itself as well as the field names. Then in app.py change num_predict from 40 to 10, rebuild, and run it. Predict whether the summary will be longer or shorter than before. The video caps the answer so that a slow model cannot ramble.

    A hint

    Use a new image tag and a new port, so your earlier container is left alone.

  4. Listen on the wrong address

    First build the image: docker build -q -t yt-api:1 . Then start a second container from it with the command overridden to run Uvicorn with --host 127.0.0.1, published on port 18001, as in the video. Predict what each of the three calls in call_api.py prints, then run python call_api.py 18001. Remove the container with docker rm -f afterwards.

    A hint

    Inside a container, 127.0.0.1 means only the container itself can connect.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

One container on one laptop calling Ollama on the same machine; no cloud deployment, sign-in, load test or scaling. host.docker.internal works out of the box with Docker Desktop; plain Docker on Linux needs an extra setting. One mistake (the bind address) is reproduced; the summary text itself is not shown or checked.

Up next in MLOps and deployment

More in MLOps and deployment

All MLOps and deployment videos

Related videos