Scenario: The Infinite Restart

The Symptom

A Java service is deployed to a new namespace. It never comes up.

kubectl get pods
# NAME                   READY   STATUS             RESTARTS      AGE
# api-6b9f7c4d8-mn2xq    0/1     CrashLoopBackOff   7 (30s ago)   6m

The developer checks logs and gets nothing:

kubectl logs api-6b9f7c4d8-mn2xq
# (empty)

An empty log plus a restart count climbing is where a lot of people get stuck — there's apparently no information to work with. There's plenty; it's just not where they looked.

The Investigation

The logs you can't see

Live logs come from the current attempt, which may have barely started — or the container may not be running at that instant at all. The error is in the run that already died:

kubectl logs api-6b9f7c4d8-mn2xq --previous

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory failed
There is insufficient memory for the Java Runtime Environment to continue.

That single flag — --previous — is the difference between "no information" and a precise answer.

Confirming with describe

kubectl describe pod api-6b9f7c4d8-mn2xq

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  7
Events:
  Warning  Unhealthy  kubelet  Liveness probe failed: Get "http://10.1.2.7:8080/healthz": dial tcp: connect: connection refused
  Normal   Killing    kubelet  Container api failed liveness probe, will be restarted

Two distinct problems, visible together:

• OOMKilled with exit 137 — the memory limit was hit.
• A liveness probe failing during startup — the application was killed before it finished booting.

The Mechanism

Why it was OOM-killed

resources:
  limits:
    memory: 512Mi

The JVM was sizing its heap from what it believed was available memory, then allocating beyond the container's 512Mi cgroup limit. From the resources guide: memory is a hard wall — cross it and the kernel terminates the process. The JVM believed it was well within budget; the cgroup disagreed.

Why the liveness probe made it worse

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3

That gives roughly 40 seconds before a restart. A Spring Boot application with a real dependency graph takes considerably longer. So even without the memory problem, this configuration kills the application mid-boot, forever.

Two independent faults producing one symptom — which is why the fix needs both halves.

The Fix

1. Right-size memory and tell the JVM about it:

resources:
  requests:
    memory: 768Mi
  limits:
    memory: 1Gi
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"

MaxRAMPercentage sizes the heap as a fraction of the container's limit, leaving headroom for metaspace, thread stacks and native memory — the parts people forget count toward the cgroup limit.

2. Add a startup probe so liveness stops interfering:

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30        # up to 150s to boot
 
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3         # once up, detects a hang in ~30s

While the startup probe runs, liveness is suspended entirely. Slow boot tolerated, fast failure detection retained afterwards.

Verifying

kubectl rollout status deploy/api
kubectl get pod -l app=api -o jsonpath='{.items[0].status.containerStatuses[0].lastState}'   # should be empty
kubectl top pods -l app=api          # actual usage vs the new limit

Then confirm the headroom is real by watching usage under load rather than assuming the new number is right.

In an Interview

"A pod is in CrashLoopBackOff. Walk me through your debugging." This is probably the most common Kubernetes interview question, and it's testing method rather than trivia.

What's being tested

• Do you have a systematic order? Candidates who start guessing at causes do worse than those who name a sequence.
• Do you know --previous? It's the specific detail that separates people who've debugged this from people who've read about it.
• Can you read an exit code? 137 → SIGKILL → memory limit or a failed shutdown.

How to answer

Give the order, then the reasoning:

"First kubectl describe pod for the events and the Last State — that tells me whether it's OOMKilled, a probe failure, a missing ConfigMap, or a bad command. Then kubectl logs --previous, because the current attempt often has no output. Those two usually settle it."

Then interpret: "Exit 137 with OOMKilled means the memory limit. Exit 137 shortly after a stop means it ignored SIGTERM. A probe failure in the events means the probe is misconfigured or the app genuinely can't start."

Follow-ups to expect

"Why would a JVM get OOM-killed inside its limit?" Heap is not the whole picture — metaspace, thread stacks, code cache and native allocations all count toward the cgroup limit. Older JVMs also sized the heap from host memory rather than the cgroup. MaxRAMPercentage is the modern control.

"How does a startup probe differ from initialDelaySeconds?" initialDelaySeconds forces one tradeoff for the whole lifetime — a long delay means slow failure detection forever. A startup probe separates the two: a generous boot budget, then tight liveness checks once running.

"When would you not add a liveness probe at all?" If the application exits on unrecoverable errors, Kubernetes restarts it anyway. Liveness earns its place only when a process can hang without exiting — and a badly configured one causes more outages than it prevents.

Next Scenario

This failure was loud once you looked in the right place. The next one is quieter — an intermittent error nobody could reproduce, caused by two pods in the same Deployment running different configuration.