Scenario: The Stack That Only Failed in CI

The Symptom

The integration suite passes locally. Every time. Nobody has seen it fail on a laptop in months.

On CI it fails roughly one run in four, always the same way:

API container exited (1)
Error: connect ECONNREFUSED 172.19.0.3:5432

Re-running the job usually makes it green. So the team does what teams do: they add a retry to the pipeline, and "just re-run it" becomes part of onboarding.

Six weeks later a genuine bug ships, because a real failure was re-run three times and then waved through as "the flaky one".

Reading "Intermittent" Correctly

"Flaky" isn't a diagnosis, it's a description of a symptom. A test that fails non-deterministically is usually one of three things:

1. A race condition — two things happen concurrently and the order isn't guaranteed.
2. Shared mutable state — leftover data from a previous run.
3. A genuine timing-sensitive bug in the code under test.

The distinguishing evidence here points hard at the first. The failure is environment-dependent (CI fails, laptops don't) and always the same error (the database connection, never anything else).

Environment-dependent timing is the signature of a race. A developer laptop is warm — images cached, filesystem cache primed, nothing else competing. A CI runner is cold, shares CPU with other jobs, and pulls images fresh. If something in your startup depends on one thing finishing before another starts, the cold environment is where it loses.

The Investigation

The Compose file:

services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
 
  api:
    build: .
    depends_on:
      - db
    environment:
      DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD}@db:5432/myapp

depends_on: [db] is right there. It looks like it handles exactly this.

It does not.

Watch the ordering on a failing run:

docker compose up

Container myapp-db-1   Created
Container myapp-db-1   Started        # <- 0.4s
Container myapp-api-1  Created
Container myapp-api-1  Started        # <- 0.6s  ...api starts connecting now
myapp-db-1   | The files belonging to this database system will be owned by "postgres"
myapp-db-1   | ...
myapp-db-1   | database system is ready to accept connections   # <- 3.2s
myapp-api-1  | Error: connect ECONNREFUSED

The database container started in under a second. Postgres was ready to accept connections at 3.2 seconds. The API tried to connect somewhere in that 2.6-second window and exited.

The Mechanism

Short-form depends_on is equivalent to condition: service_started. It waits for the container to be started — not for the application inside it to be ready.

Container started and application ready are two different events, and the gap between them is where the race lives. On a warm laptop that gap is small and the API's own startup takes long enough to cover it. On a cold CI runner the gap widens — image pull, slower disk, contended CPU — and sometimes the API gets there first.

Nothing is broken. The configuration says "start the database container first", and that's precisely what happens.

The Fix

Two halves, and both matter.

1. Give the dependency a health check so there's a readiness signal to wait for:

  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 10s

2. Wait for that signal rather than for container start:

  api:
    build: .
    depends_on:
      db:
        condition: service_healthy

The three available conditions:

Sequencing migrations

The third condition solves the related ordering problem — running migrations before the application starts:

  migrate:
    build: .
    command: ["npm", "run", "migrate"]
    depends_on:
      db:
        condition: service_healthy
 
  api:
    build: .
    depends_on:
      migrate:
        condition: service_completed_successfully

Database healthy → migrations run to completion → API starts. Deterministic.

Retry Is Complementary, Not a Substitute

A reasonable objection: shouldn't the application just retry?

Yes — and for a different reason. Your application should survive losing its database connection at any point, not only at startup. Networks partition, databases fail over, connection pools get reset. Retry with backoff is correct production behaviour regardless of Compose.

But retry is not a replacement for correct ordering:

• Retry masks the ordering problem instead of fixing it, so startup time becomes unpredictable.
• It doesn't help for one-shot jobs like migrations, where "retry later" isn't available.
• It hides genuine misconfiguration — an unreachable database looks identical to a slow one until the retries run out.

Do both. Correct ordering makes startup deterministic; retry makes the running system resilient.

In an Interview

This scenario is valuable because it's as much about debugging methodology as about Docker.

What's being tested

• Do you treat "flaky" as a symptom rather than a category? The strongest signal you can give is refusing to accept intermittent as an explanation.
• Do you know what depends_on actually guarantees? A very common misconception, and a crisp differentiator.
• Do you reason about why environments differ? Warm-versus-cold explains a huge share of works-locally-fails-in-CI problems.

How to answer

Start with the classification: "Intermittent and environment-dependent, with the same error every time, points at a race rather than genuine randomness. The question is what's racing."

Then the mechanism: depends_on in short form waits for container start, not application readiness, so there's a window where the dependency's container exists but the service inside isn't accepting connections yet. A cold CI runner widens that window.

Then the fix — health check plus condition: service_healthy — and the nuance that application-level retry is complementary rather than an alternative.

Follow-ups to expect

"Why does it pass locally?" Warm caches, no CPU contention, no image pull. Startup timings compress and the API's own boot covers the gap. The race is always there; the laptop just wins it.

"How does this translate to Kubernetes?" Directly, and it's worth raising unprompted. Readiness probes serve the same purpose — a pod isn't added to a Service's endpoints until it's ready. Kubernetes has no depends_on equivalent for ordering across workloads, so the pattern becomes init containers, Jobs for migrations, and applications that tolerate dependencies being temporarily unavailable.

"What if the health check is wrong?" A check that only confirms a port is open can pass before the service is usable — pg_isready is meaningful, a TCP connect to 5432 much less so. And give it a start_period, or a slow-booting service gets marked unhealthy before it ever finished starting.

Phase 6 in One Table

Six incidents, and the check that would have caught each before production:

Every one of these is a command that takes seconds. None of them was run, because in every case the configuration looked correct and nothing reported an error.

That's the thread through this whole roadmap: a build that succeeds is not a build that's right. Verifying behaviour — what the image contains, where the data lands, how long the stop takes, which platforms were published — is the habit that separates knowing Docker from being able to run it.

Where to Go Next

You've now got the mechanics and the war stories. Running containers across a fleet — scheduling, self-healing, rolling updates, service discovery at scale — is orchestration, and that's what the Kubernetes roadmap picks up. Several scenarios here have direct sequels there: readiness probes generalise the depends_on problem, terminationGracePeriodSeconds generalises the SIGTERM problem, and mixed-architecture node pools make exec format error a scheduling concern.