Scenario: The Database That Forgot Everything

The Symptom

A routine Tuesday. Postgres 17.2 has a patch release; the team bumps the tag and redeploys.

docker compose down
docker compose up -d

The stack comes up. Postgres is healthy. The application starts. And every table is empty.

Not corrupted — empty. A freshly initialised database, as though it had never been used. Three weeks of staging data, gone in the time it took to pull a new image.

The part that makes this scenario worth studying: the Compose file declared a volume, and everyone had reviewed it.

The Investigation

Here's the configuration, which had passed review more than once:

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/data
 
volumes:
  pgdata:

A named volume, declared at the bottom, mounted in the service. It looks right.

First check — does the volume exist and is it attached?

docker volume ls
# DRIVER    VOLUME NAME
# local     myapp_pgdata
 
docker inspect --format '{{json .Mounts}}' myapp-db-1 | jq

[
  {
    "Type": "volume",
    "Name": "myapp_pgdata",
    "Source": "/var/lib/docker/volumes/myapp_pgdata/_data",
    "Destination": "/data",
    "RW": true
  }
]

The volume exists. It's attached. It's writable. Everything confirms the configuration is doing exactly what it says.

Now the question that actually matters:

docker exec myapp-db-1 ls -la /data
# total 8
# drwxr-xr-x 2 root root 4096 Aug 21 09:14 .
# drwxr-xr-x 1 root root 4096 Sep 13 11:02 ..

Empty. The volume has been mounted for three weeks and contains nothing.

docker exec myapp-db-1 df -h /var/lib/postgresql/data
# Filesystem      Size  Used Avail Use% Mounted on
# overlay          59G   12G   44G  22% /

There it is. Postgres writes to /var/lib/postgresql/data, and that path reports the overlay filesystem — the container's writable layer. The volume was mounted at /data, a directory Postgres never touches.

The Mechanism

Two facts from earlier phases, colliding:

1. A container's writable layer is deleted when the container is removed. docker compose down removes containers.
2. Docker will mount a volume anywhere you tell it to. It has no idea what the application inside intends to use.

So for three weeks Postgres wrote happily into its writable layer. Everything worked. Every query returned correct results. The volume sat beside it, mounted and empty, providing exactly the reassurance needed to stop anyone from checking.

The Fix

One line:

    volumes:
      - pgdata:/var/lib/postgresql/data

The correct path comes from the image's documentation, not from inference. Common ones worth knowing:

Verifying It This Time

The whole lesson is that reading the config isn't verification. Three commands are:

# 1. Is the mount at the path the application uses?
docker inspect --format '{{json .Mounts}}' myapp-db-1 | jq
 
# 2. Is that path actually on the volume, not the overlay?
docker exec myapp-db-1 df -h /var/lib/postgresql/data
 
# 3. Is real data landing there?
docker exec myapp-db-1 ls -la /var/lib/postgresql/data

Command 2 is the decisive one. If df reports overlay, your data is in the writable layer regardless of what the Compose file says.

Then prove it end to end — the test that would have caught this in five minutes on day one:

# write something, destroy the container, bring it back, look for it
docker compose exec db psql -U postgres -c "CREATE TABLE canary(id int); INSERT INTO canary VALUES (1);"
docker compose down
docker compose up -d
docker compose exec db psql -U postgres -c "SELECT * FROM canary;"

In an Interview

"How do you persist data in Docker?" sounds like a definitions question. The answer that stands out treats it as a failure-modes question.

What's being tested

• Do you know the writable layer is ephemeral? This is the foundation everything else rests on.
• Can you distinguish volumes from bind mounts and say when each applies?
• Do you verify, or do you assume? The scenario above is exactly the gap between the two.

A strong answer, in shape

Start with the mechanism: a container's writable layer is discarded when the container is removed, so anything that must survive lives outside it. Then the options — named volumes for state (Docker-managed, portable, no host-path dependency), bind mounts for development source (host directory mapped in, immediate visibility), tmpfs for data that should never touch disk.

Then the part most candidates skip: how you confirm it works. Mount at the path the application actually uses, check with docker inspect and df from inside, and run a destroy-and-recreate test before trusting it.

Follow-ups to expect

"Volume or bind mount for production?" Named volume. A bind mount ties the deployment to a specific host path with specific ownership — which breaks on a Linux host where container UID and host UID differ, a problem Docker Desktop hides during development.

"How would you back up a volume?" Mount it into a throwaway container alongside a host directory and archive it. Then the point worth making unprompted: for a database, a logical dump (pg_dump) is usually safer than a file-level copy, which can capture half-written state. And a backup you have never restored is a hypothesis, not a backup.

"What's the risk of docker compose down -v?" It deletes named volumes. It's a reasonable way to reset a dev environment and a catastrophe run from habit against something you cared about. Plain down leaves them intact.

Next Scenario

Data now survives. The next scenario is about a number — every deploy stalling for exactly ten seconds — and why that precise figure hands you the diagnosis before you look at anything else.