Containers for AI Labs: Docker and Compose from Zero

Why every lab here starts with docker compose up

The most common reason a beginner abandons an AI course is not the maths. It is the twenty minutes on page one where the install fails — a Python version mismatch, a compiler missing, a CUDA wheel that will not resolve, a package that needs a package that needs a package. By the time it works, the appetite is gone. Dependency hell has ended more AI careers than any concept in machine learning ever has.

Containers exist to make that failure impossible. A container ships the entire filesystem the software needs: the exact Python, the exact libraries, the exact system packages, already built and already known to work. You do not install a lab. You start it.

There is a second reason, which matters more once you start producing results other people will act on. A finding is only real if someone else can reproduce it. If you show that a model can be talked past its guardrail, and your evidence is "it worked on my laptop", the finding is worth very little. If your evidence is a directory with a compose.yaml, a pinned image, and a test that fails on the unpatched configuration and passes on the patched one, that is a finding an engineer can act on before lunch. Containers are the difference between an anecdote and evidence.

Step 1 — Check what you have

docker version           # client and server (daemon) versions
docker compose version   # note: a space, not a hyphen — Compose v2 is a plugin
docker run --rm hello-world

If docker version prints the client but errors on the server, the Docker daemon is not running — start Docker Desktop, or sudo systemctl start docker on Linux. If docker compose version fails but docker-compose (with a hyphen) works, you are on the old standalone v1; every command on this page uses v2 syntax, so upgrade before continuing.

Step 2 — Images and containers: the distinction everything else depends on

An image is a frozen, read-only filesystem plus some metadata about how to start it. It is a build artifact, like a .zip with instructions attached. It does not run, it does not change, and it is identified by a name and tag such as python:3.12-slim.

A container is a running process — or a stopped one — that uses an image as its starting filesystem, with a thin writable layer on top for anything it changes. Start ten containers from one image and you have ten independent writable layers over one shared read-only base. Delete the container and the writable layer goes with it. The image is untouched.

The analogy people reach for is "class and object", and it is close enough to be useful. The place it misleads is here: a container is not a virtual machine. There is no second kernel. A container is a normal Linux process on the host, fenced off with namespaces (so it sees its own filesystem, network and process list) and constrained with cgroups (so it can be capped on CPU and memory). This has three consequences you should hold on to:

1. Containers start in milliseconds, because there is no operating system to boot.
2. A Linux container needs a Linux kernel. On macOS and Windows, Docker runs one inside a VM for you — which is why file access across the boundary is slower and why the memory ceiling in Step 1 exists.
3. Isolation is real but shallow. It is a strong operational boundary and a weak security boundary. We return to this in Step 10, because it is exactly the assumption people get wrong when they let an AI agent run code.

Step 3 — Your first containers

docker run --rm -it ubuntu:24.04 bash     # interactive shell, deleted on exit
docker run --rm ubuntu:24.04 echo hi      # run one command, print, exit
docker ps                                 # containers running right now
docker ps -a                              # including stopped ones
docker stop <name>                        # polite shutdown
docker rm <name>                          # delete a stopped container
docker exec -it <name> bash               # get a shell inside a RUNNING container

Try this deliberately: start ubuntu:24.04, create a file, exit, and start it again. The file is gone. This surprises everyone once and nobody twice. Containers are disposable by design; anything you want to keep has to live in a volume or a bind mount, which is Step 5.

Step 4 — Images, tags and pinning

docker pull python:3.12-slim     # download an image
docker images                    # what is on disk
docker image inspect python:3.12-slim | head -40
docker history python:3.12-slim  # the layers it is built from

A tag is a moving label, not a version. :latest is the worst of them — it means "whatever the publisher last pushed", so the same command produces different software on different days, which is the exact problem containers were supposed to solve. The safest reference is a digest, a content hash that can only ever mean one image:

# after pulling, record the digest you actually got
docker image inspect --format '{{index .RepoDigests 0}}' ollama/ollama:latest
# then pin it in your compose file, e.g. image: ollama/ollama@sha256:<digest>

We do not print a specific digest or version number in this workbook, because by the time you read it, it would be stale — and a stale pin published as fact is worse than no pin. Pull the image, read your own digest, commit that. For a graded lab, the digest is part of the evidence: it is what makes "it passed" mean something six months later.

Step 5 — Volumes: where data survives

Two mechanisms, used for different things.

A bind mount maps a directory on your machine into the container. Use it for your code and your results, so you can edit files in your normal editor and the container sees them instantly.

A named volume is storage Docker manages for you, outside any container. Use it for caches and databases — things you want to survive but never edit by hand. The model files Ollama downloads are the canonical case: they live at /root/.ollama in the container, and if that is not a volume you will re-download gigabytes every time you rebuild.

docker run --rm -v "$PWD":/work -w /work python:3.12-slim python /work/eval.py   # bind mount
docker volume create ollama-models
docker run --rm -v ollama-models:/root/.ollama ollama/ollama ollama list        # named volume
docker volume ls
docker volume inspect ollama-models

Step 6 — Ports and container networking

A container has its own network namespace. A server listening inside it is unreachable from your laptop until you publish the port.

docker run --rm -p 11434:11434 ollama/ollama       # hostPort:containerPort
docker run --rm -p 127.0.0.1:11434:11434 ollama/ollama   # bind to loopback only

Two rules that between them explain most beginner networking confusion:

localhost inside a container means the container. A script running in container A that calls http://localhost:11434 is calling itself, not your model. This is the single most common error in this curriculum.

Containers in the same Compose project reach each other by service name. If the service is called ollama, then from any other service in that project the address is http://ollama:11434. No published port required — publishing is only for reaching in from the host.

And note the difference between the two commands above. -p 11434:11434 binds to every interface on your machine, so an unauthenticated model server is now answering to anyone on the café Wi-Fi. -p 127.0.0.1:11434:11434 binds to loopback only. Ollama ships no authentication of its own, so the bind address is the access control. That is CY0-001 2.3 — access controls for AI systems — showing up as one line of YAML rather than as a policy document.

Step 7 — compose.yaml: the whole lab in one file

Typing long docker run commands does not scale past two containers. Compose puts the whole stack in a file you commit next to your code, and starts it with docker compose up.

Here is a complete, working lab stack: a local model, and a Python container that talks to it and runs tests. Save it as compose.yaml.

name: kalirange-substrate

services:
  ollama:
    image: ollama/ollama:latest        # pin to a digest you have pulled — see Step 4
    ports:
      - "127.0.0.1:11434:11434"        # loopback only: do not serve the local network
    volumes:
      - ollama-models:/root/.ollama    # keep the downloaded weights between runs
    environment:
      OLLAMA_KEEP_ALIVE: "5m"          # unload the model after idle, reclaim RAM
      OLLAMA_NUM_PARALLEL: "1"         # one request at a time is right for CPU-only
    healthcheck:
      # do not rely on curl being present in the image; the ollama binary is, so use it
      test: ["CMD-SHELL", "ollama list >/dev/null 2>&1 || exit 1"]
      interval: 5s
      timeout: 5s
      retries: 20
      start_period: 20s
    mem_limit: 6g                      # a cap the OS will enforce (see Step 10)
    restart: unless-stopped

  lab:
    profiles: ["tools"]                # kept out of a bare `up`; `compose run lab` enables it
    build:
      context: .
      dockerfile: Dockerfile
    depends_on:
      ollama:
        condition: service_healthy     # wait for READY, not merely for STARTED
    environment:
      OPENAI_BASE_URL: "http://ollama:11434/v1"   # service name, not localhost
      OPENAI_API_KEY: "ollama"                    # ignored by Ollama, required by clients
      MODEL: "llama3.2:1b"
    volumes:
      - ./work:/work                   # your code and results, editable from the host
    working_dir: /work
    command: ["pytest", "-q"]

volumes:
  ollama-models:

And the Dockerfile the lab service builds:

FROM python:3.12-slim

# System packages first: this layer changes almost never, so it stays cached.
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl jq ca-certificates \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /work

# Dependencies BEFORE application code. Editing eval.py must not reinstall pandas.
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["pytest", "-q"]

The comment about layer order is the single most useful thing in that file. Docker caches each instruction as a layer and reuses the cache until an instruction's inputs change — after which every later layer is rebuilt. Copy your source before installing dependencies and you reinstall the world on every one-character edit. Copy requirements.txt first and installs happen only when dependencies actually change. The same discipline is why rm -rf /var/lib/apt/lists/* sits inside the same RUN as apt-get install: a file deleted in a later layer still occupies space in the image, because layers only stack, never subtract.

Two honest notes about that file. requirements.txt is a placeholder that keeps the layer-order lesson uncluttered: the Python page shows why pip freeze > requirements.txt is not a lockfile, and replaces it with pyproject.toml plus a committed uv.lock, which is what the graded labs install from — same ordering rule, manifest and lockfile first, source last. And COPY . . is inert at run time in this stack, because Compose bind-mounts ./work over /work and the mount wins; it earns its place only when the image is run without a mount. Add a .dockerignore as well, or the build context — the repo root — quietly puts your compose.yaml and your results inside the image.

docker compose up -d ollama                            # start the model server
docker compose exec ollama ollama pull llama3.2:1b     # download the weights, once
docker compose exec ollama ollama list                 # confirm it is there

# smoke-test the API from your host (the port is published on loopback)
curl -s http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.2:1b","messages":[{"role":"user","content":"Say OK"}],"stream":false}' \
  | jq -r '.choices[0].message.content'

docker compose run --rm lab pytest -q                  # run the graded tests

The weights land in the ollama-models volume, so they survive docker compose down, image rebuilds, and everything else short of Step 9's destructive command.

Step 8 — Logs and debugging

When something does not work, the answer is almost always already printed somewhere.

docker compose ps                # what is up, and is it healthy
docker compose logs ollama       # everything that service has said
docker compose logs -f --tail=50 ollama   # follow live, last 50 lines
docker compose exec ollama bash  # get a shell inside the running container
docker compose run --rm lab bash # a throwaway shell in a NEW lab container
docker inspect <container> --format '{{.State.ExitCode}}'
docker stats                     # live CPU and memory per container

Note the difference between exec and run. exec steps into a container that is already running — right for looking at the model server. run starts a fresh one from the image — right for trying a command without disturbing anything.

Step 9 — Cleaning up (and the one command that eats your models)

Docker accumulates. Images, stopped containers, build cache and volumes all quietly consume disk until a laptop runs out during a lab.

docker compose down          # stop and remove containers + the project network
docker compose down -v       # ...AND DELETE THE NAMED VOLUMES
docker system df             # how much space is used, by category
docker container prune       # remove all stopped containers
docker image prune -a        # remove images no container is using
docker builder prune         # clear the build cache (often the biggest offender)
docker system prune -a       # all of the above, aggressively

Step 10 — The security part everyone skips

Containers are an operational boundary that people keep spending as a security boundary. Four habits, each cheap, each one closing a real hole in an AI lab.

Never mount the Docker socket into a container an AI agent can reach. -v /var/run/docker.sock:/var/run/docker.sock hands that container the ability to start any other container, including a privileged one mounting the host filesystem. That is not a container escape, it is a container escape feature. Tutorials that build "an agent that can manage its own environment" do this constantly. When the agent's instructions can be influenced by a document it reads, you have connected untrusted text to root on the host — the shape of LLM03:2026 Excessive Agency. (OWASP's Top 10 for LLM Applications is licensed CC BY-SA 4.0; the descriptions here and below are our own paraphrase.)

Cap resources. mem_limit and cpus in the Compose file, plus OLLAMA_NUM_PARALLEL: "1", keep one runaway generation from freezing your machine. In production the same control has a name: it is the mitigation for LLM06:2026 Unbounded Consumption, where an attacker sends prompts engineered to be maximally expensive. On your laptop it is the difference between a slow lab and a hard reboot.

No secrets in images. Anything COPYed or ENV-set during a build is baked into a layer, and docker history will show it to anyone who pulls the image — deleting the file in a later instruction does not remove it from the earlier layer. Secrets belong in environment variables at run time, an env_file that is in .gitignore, or a mounted file with mode 600.

Cut the network when a lab does not need it. network_mode: none on a service, or an internal: true network for the project, means a model that gets talked into exfiltrating something has nowhere to send it. This is the crispest control in the whole curriculum, and the one most often left out: an offline lab cannot leak.

Together these are CY0-001 2.2 — implementing security controls for AI systems — done as configuration rather than as prose.

How the lab for this page is graded

The lab hands you a broken compose.yaml with four seeded defects: a service published on all interfaces, a depends_on without a health condition, a Dockerfile that copies source before requirements, and a missing volume for the model cache. Grading is a hidden pytest suite that parses your compose.yaml and Dockerfile and asserts the invariants — the host binding contains 127.0.0.1, condition: service_healthy is present, the COPY requirements.txt line index is lower than the COPY . . line index, and a named volume maps to /root/.ollama. It never compares your file to a reference file, so any correct arrangement passes. A final check brings the stack up and requires one successful completion from the model. Passing prints KR{compose-up-is-the-front-door}.

📋 Cheat sheet