A Repeatable Method for Debugging Containers

Guessing Doesn't Scale

The instinct when a container misbehaves is to start changing things — add a flag, tweak the Dockerfile, restart and hope. It occasionally works, and it teaches you nothing transferable.

A fixed order works better, because it narrows the problem before you touch anything:

Most problems resolve at step 1 or 2. The value of the order is that you never skip the cheap checks to go fiddling with the expensive ones.

Step 1: State and Exit Code

docker ps -a
docker compose ps

The STATUS column carries more information than people read from it:

• Up 4 minutes (healthy) — running, health check passing.
• Up 10 seconds (health: starting) — inside its start period; not a failure yet.
• Restarting (1) 3 seconds ago — crash loop. The number in brackets is the exit code.
• Exited (137) 2 minutes ago — killed. Read on.

The exit codes from phase 1, with what to do about each:

Step 2: Logs — Including the Ones You Can't See

docker logs myapp
docker logs -f --tail 100 myapp          # follow, last 100 lines
docker logs --since 10m myapp            # recent only
docker compose logs -f api               # one service in a stack

For a container that's crash-looping, the live logs are from the current attempt, which may have barely started. The previous run holds the actual error:

docker logs --previous myapp

Step 3: Inspect — What It Actually Got

When the logs don't explain it, stop reading the command and read the container:

docker inspect --format '{{json .State}}'    myapp | jq   # exit code, OOMKilled, timestamps
docker inspect --format '{{json .Mounts}}'   myapp | jq   # what's mounted, where
docker inspect --format '{{json .Config.Env}}' myapp | jq # resolved environment
docker inspect --format '{{json .NetworkSettings.Networks}}' myapp | jq
docker inspect --format '{{json .State.Health}}' myapp | jq # probe history + failure output
docker inspect --format '{{json .HostConfig.Memory}}' myapp # the limit, in bytes

This is the step that catches the failures configuration files hide: a volume mounted at a path the application doesn't use, an environment variable that resolved to an empty string, a container on a different network than you assumed.

For a stack, the Compose equivalent is docker compose config — the merged, interpolated configuration Compose will actually apply.

Step 4: Resources and Daemon Activity

docker stats                      # live CPU, memory vs limit, network, disk I/O
docker stats --no-stream          # one snapshot

The MEM USAGE / LIMIT column is the one to watch. A container sitting at 98% of its limit is about to become an exit 137.

CPU limits behave differently and are easier to miss: a container at its CPU limit isn't killed, it's throttled. The symptom is latency and timeouts with no crash and no error message — which is why "it's slow but nothing is wrong" so often turns out to be a CPU limit.

docker events --since 30m                                    # what did the daemon do?
docker events --filter 'container=myapp' --filter 'event=die'

docker events is underused. It answers "who stopped my container?" — showing whether it exited on its own, was killed by the OOM killer, or was stopped by something else.

docker top myapp        # processes inside — reveals zombies and stray children
docker history myapp:latest   # which build step made the image large
docker system df -v     # disk used by images, containers, volumes, build cache

Step 5: Looking From the Inside

docker exec -it myapp sh          # a shell in a running container
docker exec myapp env             # what the process actually sees
docker exec myapp ls -la /app/data
docker exec myapp getent hosts db # does this hostname resolve?
docker exec myapp df -h /var/lib/postgresql/data  # is that path really on a volume?

Remember from phase 1: use exec, not attach. attach connects to the main process, and Ctrl+C there sends an interrupt to your application.

When the container has no shell

Phase 2's tradeoff arrives: distroless images have no sh, so docker exec -it myapp sh fails with "executable file not found".

Three ways through:

docker debug attaches a toolbox — with a shell and utilities like curl and vim — to a running container without modifying it. Changes are temporary and the image stays lean.

docker debug myapp

A debug stage in your Dockerfile. Using --target from phase 2, keep a variant of the same application on a fuller base:

FROM gcr.io/distroless/nodejs22-debian12 AS runtime
COPY --from=build /app/dist /app/dist
CMD ["/app/dist/server.js"]
 
FROM node:22-slim AS debug
COPY --from=build /app/dist /app/dist
CMD ["node", "/app/dist/server.js"]

docker build --target debug -t myapp:debug .

A sidecar sharing the namespaces. Universal, and works anywhere:

docker run -it --rm \
  --pid=container:myapp \
  --network=container:myapp \
  nicolaka/netshoot

That container joins the target's PID and network namespaces, so you can inspect its processes and network from a container that does have tools.

Errors Worth Recognising

exec format error — architecture mismatch. An arm64 image on an amd64 host or vice versa. The multi-platform build from phase 2 is the fix.

connection refused between containers — almost always one of two things. Either they aren't on a shared user-defined network (check with getent hosts), or the application bound to 127.0.0.1 instead of 0.0.0.0. Both are covered in the networking guide.

no such file or directory on a binary that exists — usually a dynamic linking failure rather than a missing file: a glibc-built binary on musl-based Alpine. The base-image guide covers this tradeoff.

permission denied on a mounted directory — UID mismatch on a Linux bind mount. Try -u "$(id -u):$(id -g)".

toomanyrequests / pull rate limit — registry rate limiting, common in CI on shared IPs. Authenticate, or pull through a registry mirror.

no space left on device — check docker system df before blaming the host disk. Old images, stopped containers and build cache accumulate silently.

Why This Is the Skill That Matters Now

Every command here observes actual behaviour. That's what makes this the most durable skill in the roadmap.

You will increasingly work with Dockerfiles, docker run lines and Compose files that came from an AI assistant, a search result, or a colleague's snippet. They're frequently good, and using them is sensible. But the check "did it start without error?" is much weaker than it feels — as phases 2 and 3 showed, a container can start perfectly while mounting a volume nowhere useful, leaking a token into image history, or silently ignoring SIGTERM.

So make the loop deliberate: run it, then verify the specific thing you're relying on.

• Relying on persistence? docker inspect the mounts, then df -h the data path from inside.
• Relying on a clean shutdown? docker stop and time it. Under a second is right; ten seconds is a bug.
• Relying on a secret not leaking? docker history the image and look.
• Relying on service discovery? getent hosts from inside the container.

Each takes seconds and checks the actual property, not your reading of the config.

Phase 4 in Three Sentences

1. Compose's version: key is obsolete and docker-compose v1 is dead — use docker compose, and delete the version line.
2. Plain depends_on waits for container start, not readiness; pair a healthcheck with condition: service_healthy.
3. Debug in a fixed order — state, logs, inspect, resources, inside — and verify behaviour rather than re-reading configuration.

What's Next

The final phase covers what changes when an image leaves your laptop: dropping root and setting resource limits, handling secrets at build and run time, and knowing what's actually inside what you ship — scanning, SBOMs and build provenance.