Disruption, Draining and Graceful Termination

Pods Are Meant to Be Disturbed

Nodes get drained for upgrades. Autoscalers remove capacity. The scheduler rebalances. Hardware fails. Pods being terminated and replaced is routine, not exceptional — and applications that assume otherwise break in ordinary operation.

Kubernetes distinguishes two kinds:

• Involuntary disruption — a node dies, hardware fails, the kernel OOM-kills something. No warning.
• Voluntary disruption — a drain for maintenance, a rollout, a scale-down. Initiated deliberately, and therefore controllable.

PodDisruptionBudgets govern the second kind.

PodDisruptionBudgets

A PDB tells the cluster the minimum availability to respect during voluntary disruption:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2          # or: maxUnavailable: 1
  selector:
    matchLabels:
      app: api

Now kubectl drain will evict API pods only while at least 2 remain available. If evicting one would drop below that, the eviction is refused and the drain waits.

Express it either way:

• minAvailable: 2 — an absolute floor.
• minAvailable: 50% — proportional, so it scales with the Deployment.
• maxUnavailable: 1 — often more intuitive: take out one at a time.

A PDB constrains voluntary disruption only. If the node catches fire, your pods go regardless — a PDB is not an availability guarantee, it's a guardrail on deliberate operations.

The Termination Sequence

Here's the part that decides whether your deploys drop requests, and it contains a subtlety most people get wrong.

When a pod is deleted:

1. The pod is marked for deletion with a deletion timestamp.
2. Two things then happen concurrently:
   • The kubelet begins graceful shutdown — running any preStop hook, then sending SIGTERM to PID 1 in each container.
   • The control plane evaluates removing the pod from EndpointSlices, which propagates to kube-proxy on every node.
3. The kubelet waits up to terminationGracePeriodSeconds (default 30).
4. Anything still running gets SIGKILL.

The preStop Fix

The remedy is a preStop hook that simply waits, delaying SIGTERM long enough for endpoint removal to propagate:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]

The sequence becomes:

1. Pod marked for deletion. Endpoint removal starts propagating.
2. preStop sleeps for 10 seconds — the pod is still serving normally during this.
3. kube-proxy everywhere has stopped sending new traffic.
4. SIGTERM arrives; the application drains its remaining in-flight requests.
5. Clean exit, well within the grace period.

Handling SIGTERM

The hook buys time; your application still has to use it:

const server = app.listen(3000);
 
process.on("SIGTERM", () => {
  server.close(() => {      // stop accepting new, finish in-flight
    db.end();
    process.exit(0);
  });
});

Two failure modes worth naming:

No handler. The default action for SIGTERM terminates the process immediately — in-flight requests are dropped. Better than SIGKILL, still lossy.

The signal never arrives. If your container uses shell form in its Dockerfile (CMD node server.js), PID 1 is /bin/sh, which doesn't forward signals. Your handler never runs, and the container is SIGKILLed at the end of the grace period. This is the Docker roadmap's ten-second-deploy scenario, and in Kubernetes it costs you 30 seconds per pod plus every in-flight request. Use exec form: CMD ["node", "server.js"].

kubectl exec api-7d4f8b9c5-x2kfp -- ps -ef | head -3   # is your app PID 1?

Node Drains

kubectl cordon ip-10-0-1-42                          # stop new scheduling
kubectl drain ip-10-0-1-42 --ignore-daemonsets --delete-emptydir-data
kubectl uncordon ip-10-0-1-42                        # back into service

drain cordons the node and then evicts its pods, respecting PDBs.

The flags exist because two things always get in the way:

• --ignore-daemonsets — DaemonSet pods are immediately recreated on the same node by design, so drain refuses to proceed without acknowledging them.
• --delete-emptydir-data — emptyDir data is lost when the pod moves, so drain requires explicit confirmation.

A drain that hangs is nearly always a PDB it cannot satisfy:

kubectl get pdb -A
kubectl describe pdb api        # ALLOWED DISRUPTIONS: 0 is your answer

A Production-Ready Shape

spec:
  replicas: 3
  template:
    spec:
      terminationGracePeriodSeconds: 60
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: api }
      containers:
        - name: api
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: api }

Spread across zones, one pod disruptable at a time, a preStop delay covering endpoint propagation, a grace period comfortably longer than both the hook and the drain, and a readiness probe gating traffic.

What's Next

Phase 6 is the final one: securing the cluster with RBAC and Pod Security Admission, managing manifests at scale with Helm and Kustomize, and building the debugging method that ties the whole roadmap together.