Scenario: The Zero-Downtime Deploy That Wasn't

The Symptom

Every production deploy produces a burst of 502s. Small — a few hundred errors out of hundreds of thousands of requests, lasting under a minute — but consistent, and visible on the error dashboard as a spike at exactly the time of each rollout.

The team had done the reading. They had:

• A readiness probe gating traffic
• maxUnavailable: 0 so capacity never dips
• A SIGTERM handler that drains connections
• A 60-second terminationGracePeriodSeconds

Every piece of standard zero-downtime advice was in place. The 502s continued.

The Investigation

Application logs during a deploy showed graceful shutdown working exactly as designed:

14:32:01  SIGTERM received, draining connections
14:32:01  server.close() — no longer accepting new connections
14:32:03  in-flight requests complete, exiting cleanly

Two seconds, clean exit. Nothing wrong.

But the gateway's access log told a different story:

14:32:02  502  GET /api/orders   upstream 10.1.4.22:8080
14:32:02  502  GET /api/users    upstream 10.1.4.22:8080
14:32:03  502  POST /api/cart    upstream 10.1.4.22:8080

10.1.4.22 is the pod that received SIGTERM at 14:32:01. Traffic was still being routed to it one to two seconds after it had stopped accepting connections.

The pod did nothing wrong — it was told to shut down and did so correctly. Something was still sending it traffic.

The Mechanism

Here's the detail that makes this scenario worth studying, because it contradicts what most people assume the termination order is.

When a pod is deleted, two things happen concurrently:

• The kubelet begins graceful shutdown — preStop hook, then SIGTERM.
• The control plane evaluates removing the pod from EndpointSlices.

The Kubernetes documentation states it directly: "At the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice objects."

At the same time — not before.

And endpoint removal isn't instantaneous once it starts. It has to propagate:

Every hop adds latency: the control plane writes the EndpointSlice, kube-proxy on each node watches and reprograms, and the gateway refreshes its own view. That's typically one to two seconds — matching the 502 window exactly.

The application's shutdown was too fast. It stopped accepting connections while the cluster was still directing traffic to it.

The Fix

A preStop hook that does nothing but wait:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]

The preStop hook runs before SIGTERM is sent. So the sequence becomes:

1. Pod marked for deletion; endpoint removal begins propagating.
2. preStop sleeps 10 seconds — the pod is still serving normally throughout.
3. Every kube-proxy and the gateway have stopped routing to it.
4. SIGTERM arrives; the handler drains whatever is left in flight.
5. Clean exit, comfortably inside the grace period.

Confirming PID 1

Worth checking while you're here, because it produces the same symptom for a different reason:

kubectl exec api-7d4f8b9c5-x2kfp -- ps -ef | head -3

If PID 1 is /bin/sh rather than your application, the Dockerfile used shell form (CMD node server.js), and sh doesn't forward signals. Your handler never runs at all, and every pod is SIGKILLed after the full grace period. Exec form — CMD ["node", "server.js"] — is the fix.

Verifying

The only real proof is measurement under load:

# generate steady traffic
hey -z 3m -c 20 https://api.example.com/health &
 
# roll the deployment while it runs
kubectl rollout restart deploy/api
kubectl rollout status deploy/api

Zero non-200 responses through a complete rollout is the result you want. Anything else means the window is still open — either the sleep is too short for your propagation time, or the signal isn't reaching the application.

In an Interview

"How do you achieve zero-downtime deployments in Kubernetes?" is a standard question with a standard answer — readiness probes, rolling updates, graceful shutdown. Going one level deeper is what distinguishes you.

What's being tested

• Do you know the termination sequence? Specifically that endpoint removal is concurrent with SIGTERM.
• Can you explain why preStop exists? Most people know the pattern without knowing the reason.
• Do you verify under load? Claiming zero-downtime without measuring it is the common weakness.

How to answer

Give the standard pieces, then the part most people miss:

"Readiness probes gate traffic, maxUnavailable: 0 preserves capacity, and a SIGTERM handler drains in-flight requests. But those alone still drop requests, because endpoint removal happens concurrently with SIGTERM rather than before it — so for a second or two, kube-proxy on other nodes is still routing to a pod that has begun shutting down. A preStop hook that sleeps for a few seconds delays SIGTERM until propagation completes, and the pod keeps serving throughout."

Then close on verification: "And I'd confirm it by running continuous load through a rollout and checking for non-200s, rather than assuming the configuration is sufficient."

Follow-ups to expect

"How long should the preStop sleep be?" Long enough for endpoint propagation, typically 5–15 seconds depending on cluster size and your ingress implementation. Measure it — roll under load with different values and find where errors stop.

"What if the app doesn't receive SIGTERM at all?" Check PID 1. Shell form in the Dockerfile puts /bin/sh there, and it doesn't forward signals. Same symptom, different root cause, and kubectl exec -- ps -ef distinguishes them instantly.

"How does this relate to node drains?" Identically — a drain evicts pods, which is the same termination path. So the preStop hook protects cluster upgrades and autoscaler scale-downs too, not just deploys. PDBs control how many pods go at once; preStop controls whether each one goes cleanly.

Phase 7 in One Table

Six incidents, and the check that would have caught each:

Notice how many resolve at kubectl describe. The events attached to an object are where controllers record what they tried and why it didn't work — and in four of six cases, the whole answer was sitting there before anyone opened a dashboard.

That's the thread through both roadmaps: accepted is not achieved, and running is not working. kubectl apply reports a successful write to etcd. Everything after that is asynchronous, and verifying it means observing behaviour — endpoints, events, actual resolved config, a load test through a real rollout — rather than re-reading the manifest you already believed was correct.

Where to Go Next

You've covered container fundamentals through cluster operations. Natural directions from here: GitOps with Argo CD or Flux, extending reconciliation to deployment itself; service meshes for L7 traffic policy and mTLS, which pick up where NetworkPolicy's layer-3/4 model stops; operators and custom controllers, writing your own reconciliation loops; and observability with Prometheus, Grafana and OpenTelemetry, which is what turns the debugging commands in this roadmap into signals you see before users do.