Debugging and Cluster Operations

A fixed triage order, what each failure state actually means, and how to get a shell into a distroless pod that doesn't have one.

September 13, 2026

kubernetesdebuggingkubectl-debugeventsephemeral-containerscrashloopbackoffupgrades

Stop Guessing, Start Triaging

Kubernetes gives you a lot of places to look, which is exactly why an unstructured approach wastes time. Use a fixed order:

Most problems resolve at step 2. The value of the order is that you never skip the cheap checks.

Step 1: State

bash

kubectl get pods -o wide
kubectl get pods -w                    # watch changes live
kubectl get all -n production

Read the columns carefully — READY 0/1 with STATUS Running is a readiness failure, and that pod is receiving no traffic.

Step 2: describe — Where the Answer Usually Is

bash

kubectl describe pod api-7d4f8b9c5-x2kfp

Three sections repay attention:

Events (at the bottom) — the scheduler, kubelet and controllers all record here. Scheduling failures, image pull errors, probe failures, volume problems.
State / Last State — including Reason: OOMKilled and Exit Code: 137.
Conditions — Ready, ContainersReady, PodScheduled.

bash

kubectl get events --sort-by=.lastTimestamp              # cluster-wide, newest last
kubectl get events --field-selector type=Warning -A

⚠️

Events are garbage collected, typically after about an hour. A failure from this morning may have no events left by afternoon — which is one of the strongest arguments for shipping events to a monitoring system. "There are no events" doesn't mean nothing happened.

Step 3: Logs

bash

kubectl logs api-7d4f8b9c5-x2kfp
kubectl logs api-7d4f8b9c5-x2kfp --previous        # the crashed attempt
kubectl logs api-7d4f8b9c5-x2kfp -c sidecar        # a specific container
kubectl logs -l app=api --tail=50 --prefix         # all pods matching a label
kubectl logs api-7d4f8b9c5-x2kfp -f --since=10m

--previous is the one people forget. For a crash-looping pod, the current attempt may have produced nothing — the actual error is in the run that already died.

Step 4: From the Inside

bash

kubectl exec -it api-7d4f8b9c5-x2kfp -- sh
kubectl exec api-7d4f8b9c5-x2kfp -- env
kubectl exec api-7d4f8b9c5-x2kfp -- df -h /var/lib/postgresql/data
kubectl exec api-7d4f8b9c5-x2kfp -- getent hosts db.production

When the image has no shell

A distroless image has no sh, so kubectl exec -it ... -- sh fails. Ephemeral containers solve this — kubectl debug attaches a temporary container sharing the target pod's namespaces:

bash

# Attach a debug container to a running pod
kubectl debug -it api-7d4f8b9c5-x2kfp --image=nicolaka/netshoot --target=api
 
# Copy a crashed pod with a shell as its entrypoint, leaving the original untouched
kubectl debug api-7d4f8b9c5-x2kfp -it --copy-to=api-debug --container=api -- sh
 
# Get a shell on the node itself
kubectl debug node/ip-10-0-1-42 -it --image=busybox

--target=api shares the process namespace with that container, so you can inspect its processes and its network from a container that actually has tools.

✅

The --copy-to form is the one for a pod that won't start at all. You can't exec into a container that keeps crashing — but you can copy the pod with the command replaced by a shell, then inspect the filesystem and environment by hand to work out why the real command fails.

Reading the Common Failure States

Pending

The scheduler couldn't place it. describe tells you which filter rejected which nodes:

text

0/6 nodes are available: 3 Insufficient memory, 2 node(s) had untolerated taint, 1 node(s) had volume node affinity conflict.

Causes: requests too large for any node, an untolerated taint, unsatisfiable affinity, an unbound PVC, or WaitForFirstConsumer binding (where Pending is expected, not a fault).

ImagePullBackOff / ErrImagePull

bash

kubectl describe pod <pod> | grep -A 5 Events

Usually: a wrong tag, missing registry credentials (imagePullSecrets), a private registry, a rate limit, or an architecture mismatch — an arm64 image on amd64 nodes, which surfaces as exec format error once it does pull.

CrashLoopBackOff

The container starts and exits repeatedly. This is the backoff state, never the cause.

bash

kubectl logs <pod> --previous
kubectl describe pod <pod> | grep -A 10 "Last State"

Common causes: a missing config or Secret, a failing dependency at startup, a liveness probe too aggressive for a slow boot (use a startup probe), OOMKilled, or a bad command.

OOMKilled

bash

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Exit 137, Reason: OOMKilled. Either the limit is too low or the application leaks. kubectl top pods over time distinguishes them — a leak climbs steadily.

Evicted

The node ran out of a resource and the kubelet reclaimed it. Recall from phase 4 that BestEffort pods are evicted first — a pod with no resource requests is the first thing thrown overboard.

bash

kubectl get pods -A --field-selector status.phase=Failed
kubectl describe node <node> | grep -A 5 Conditions      # MemoryPressure, DiskPressure

Check yourself

A pod is in CrashLoopBackOff. `kubectl logs <pod>` returns nothing. What's the next command?

Cluster-Level Checks

bash

kubectl get nodes
kubectl describe node ip-10-0-1-42          # conditions, allocated vs capacity
kubectl top nodes
kubectl top pods -A --sort-by=memory
 
kubectl get pods -n kube-system             # CoreDNS, CNI, metrics-server healthy?
kubectl get --raw='/readyz?verbose'         # API server health

Node conditions are worth knowing by name: Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable. A node under DiskPressure evicts pods and refuses new ones.

Cluster Upgrades

The operational task most likely to cause a self-inflicted outage.

Removed APIs are the main hazard. Kubernetes removes deprecated API versions on a schedule, and a manifest using a removed version fails to apply after the upgrade — often discovered mid-deploy.

bash

kubectl api-resources                     # what this cluster serves
kubectl api-versions

Tools such as kubent (kube-no-trouble) and pluto scan your manifests and live cluster for APIs removed in a target version. Run one before upgrading, not after.

Other upgrade-time considerations:

Version skew. Nodes may run up to a couple of minor versions behind the control plane, but not ahead. Upgrade the control plane first.
Don't skip minor versions. Upgrade one at a time.
Runtime dependencies. Kubernetes 1.35 is the last release supporting containerd 1.x — clusters must move to containerd 2.0 or later before upgrading beyond it.
Support window. The project maintains the most recent three minor releases with roughly a year of patch support each, so falling far behind means running without security fixes.
Drain properly. Node upgrades mean drains, which means the PDBs and preStop hooks from phase 5 determine whether users notice.

✅

The upgrade checklist that prevents most incidents: back up etcd, scan for removed APIs, upgrade the control plane, upgrade nodes one at a time with proper drains, and verify workloads between each step. The scan is the part teams skip and the part that bites.

The Habit Underneath All of It

Every command in this guide observes what actually happened rather than what was supposed to. That's the thread running through the entire roadmap:

Instead of assuming	Check
The manifest applied, so it's deployed	`kubectl get` + `kubectl describe` events
The Service is configured, so traffic flows	`kubectl get endpointslices`
The volume is declared, so data persists	`kubectl exec -- df -h <data-path>`
The policy exists, so traffic is blocked	A real connectivity test from a blocked pod
The RBAC looks right	`kubectl auth can-i --list --as=...`
The chart renders what I intended	`helm template` / `kubectl diff`

Each takes seconds. Each tests the actual property rather than your reading of a file.

Check yourself

Which command gets you a shell alongside a running distroless pod that has no shell of its own?

What's Next

That completes the mechanics. Phase 7 applies them to six real incidents told end to end — the symptom, the investigation, the mechanism, and how to discuss each one in an interview.

Packaging: Helm and Kustomize

Scenario: The Infinite Restart