Scenario: The Thundering Herd

The Symptom

A marketing campaign goes out. Traffic increases roughly fourfold over two minutes.

The HPA does what it was configured to do — and then the entire site goes down, harder than it would have without autoscaling at all.

The timeline from the incident review:

14:02:10  traffic begins climbing
14:02:45  HPA scales api from 4 → 20 replicas
14:03:05  new pods Running and receiving traffic
14:03:12  database: FATAL: sorry, too many clients already
14:03:20  all API pods returning 500
14:04:00  database CPU at 100%, existing connections timing out

Ninety seconds from healthy to total outage. The autoscaler was the trigger.

The Investigation

kubectl describe hpa api

Metrics:  resource cpu on pods (as a percentage of request):  340% (850m) / 70%
Events:
  Normal  SuccessfulRescale  New size: 20; reason: cpu resource utilization above target

The HPA calculated correctly. CPU was at 340% of a 70% target, so it computed a large replica count and applied it in one step — the default behaviour permits scaling up aggressively.

The database side:

max_connections = 100

Each API pod opens a pool of 10 connections. Four pods used 40. Twenty pods wanted 200.

The pool exhaustion then cascaded: connections queued, requests timed out, the health endpoint started failing, and pods that had connections couldn't get queries through because the database was saturated.

The Mechanism

Two failures compounding.

1. Unbounded scale-up. By default the HPA can scale up very aggressively — appropriate for stateless workloads with no shared downstream, disastrous for anything backed by a finite resource.

2. No readiness gate on dependencies. The readiness probe checked only that the HTTP server was listening:

readinessProbe:
  httpGet: { path: /healthz, port: 8080 }

So each new pod was added to the Service endpoints the moment it started — before its connection pool was established. It received traffic it had no way to serve, and each pod's attempt to build a pool added pressure to an already-failing database.

The Fix

1. Bound the rate of change

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
      selectPolicy: Min
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

Now scale-up adds at most 50% more pods — and never more than 4 — per minute, taking the more conservative of the two. Growth becomes gradual enough for downstream systems to absorb.

The stabilizationWindowSeconds on scale-up also prevents reacting to a brief spike that would have resolved itself.

2. Gate readiness on actual readiness

readinessProbe:
  httpGet: { path: /ready, port: 8080 }   # verifies the DB pool is usable
  periodSeconds: 5
  failureThreshold: 3

With /ready checking that the pod can genuinely reach the database, a pod that can't get connections stays out of the Service endpoints rather than accepting traffic it will fail.

This also creates useful back-pressure: if the database is saturated, new pods don't become ready, so they don't add load.

3. Fix the connection arithmetic

A connection pooler (PgBouncer for Postgres) sits between the application and the database, multiplexing many application connections onto few database connections. The pool size stops scaling linearly with replica count.

Without one, the arithmetic must be deliberate: maxReplicas × poolSize must stay comfortably below max_connections, with headroom for migrations and admin access.

4. Scale on a better metric

CPU was a poor proxy here. Request latency or queue depth describes the actual problem, and an external or custom metric would have scaled on the signal that mattered rather than one that correlated with it.

In an Interview

Autoscaling questions usually start simple — "how does the HPA work?" — and the good answer goes straight past the formula to the failure modes.

What's being tested

• Do you understand that autoscaling has downstream effects? Most candidates describe scaling in isolation.
• Do you know the behavior field exists? A specific, current detail.
• Can you connect probes to autoscaling? They're usually taught separately and they interact directly.

How to answer

Cover the mechanism briefly — the HPA compares a metric to a target and computes replicas — then move to what matters: "The interesting part is what happens downstream. Scaling pods doesn't scale your database, so aggressive scale-up can exhaust a connection pool and turn a load spike into an outage. Two controls matter: the behavior field to bound the rate of change, and a readiness probe that verifies dependencies so new pods don't take traffic before they can serve it."

Follow-ups to expect

"Why is CPU often a bad scaling metric?" For I/O-bound workloads CPU stays flat while the backlog grows. Queue depth, latency, or in-flight requests describe demand better — available via custom or external metrics, or KEDA.

"What if scaled pods can't be scheduled?" They sit Pending until Cluster Autoscaler or Karpenter adds nodes, which takes minutes. That latency is why minReplicas and headroom matter — you can't provision capacity at the instant you need it.

"Can you run VPA alongside HPA?" Not on the same resource metric — they form a feedback loop, since VPA changes requests and HPA's utilisation is a percentage of requests. Safe only when the HPA scales on an independent metric.

Next Scenario

This incident came from too many pods. The next one is the opposite — a Deployment where two of three pods could never start at all, because of one word in a storage manifest.