Scenario: The Pod That Would Not Schedule

The Symptom

A new ML inference service is deployed to a cluster that visibly has spare capacity.

kubectl get pods -l app=inference
# NAME                        READY   STATUS    RESTARTS   AGE
# inference-6c7d9f8b5-qk2mp   0/1     Pending   0          11m

Eleven minutes, no progress. kubectl top nodes shows the cluster averaging 35% CPU and 40% memory. There is obviously room.

kubectl logs inference-6c7d9f8b5-qk2mp
# Error from server (BadRequest): container "inference" in pod "..." is waiting to start: ContainerCreating

No logs, because no container exists yet. Pending means the pod has not been assigned to a node at all — so from phase 1, the kubelet has never seen it. This is entirely a scheduler question.

The Investigation

The scheduler records exactly why it failed:

kubectl describe pod inference-6c7d9f8b5-qk2mp

Events:
  Warning  FailedScheduling  default-scheduler
    0/8 nodes are available:
      3 Insufficient memory,
      2 node(s) had untolerated taint {workload: gpu},
      2 Insufficient cpu,
      1 node(s) had volume node affinity conflict.

That message is a complete summary of the filtering stage — every node, and which filter rejected it. Read it as arithmetic: 3 + 2 + 2 + 1 = 8, every node accounted for.

Four separate reasons, which is why "the cluster has spare capacity" was misleading.

The Mechanism

"Insufficient memory" and "Insufficient cpu" — 5 nodes

The scheduler places pods using requests, not actual usage. From the resources guide: requests reserve capacity, and the scheduler sums the requests of everything already on a node.

resources:
  requests:
    cpu: "4"
    memory: 16Gi

kubectl top nodes shows usage — 35%. The scheduler sees reservations, which were far higher because other workloads had over-requested.

kubectl describe node ip-10-0-1-42 | grep -A 8 "Allocated resources"

  Resource           Requests      Limits
  cpu                7200m (90%)   14 (175%)
  memory             28Gi (87%)    40Gi (125%)

90% of CPU reserved, while actual usage sat near 35%. There was no room for a 4-core request.

"Untolerated taint" — 2 nodes

The GPU nodes were tainted to keep general workloads off them:

kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

The pod needed a GPU but had no toleration, so it was excluded from exactly the nodes it belonged on.

"Volume node affinity conflict" — 1 node

A PVC had already been provisioned in us-east-1a with volumeBindingMode: Immediate. The remaining candidate node was in us-east-1b, and a zonal disk cannot cross zones — so that node was filtered out too.

The Fix

1. Tolerate the GPU taint and target those nodes:

spec:
  tolerations:
    - key: workload
      operator: Equal
      value: gpu
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values: [gpu]

Both halves are needed: the toleration permits scheduling onto tainted nodes, and node affinity directs the pod there. A toleration alone would let the pod land on an ordinary node instead.

2. Right-size the request from evidence rather than a guess — the measurement method from the resources guide.

3. Fix the volume binding mode so storage follows scheduling rather than constraining it:

volumeBindingMode: WaitForFirstConsumer

4. Address the broader over-requesting with a LimitRange supplying sane defaults and a ResourceQuota making each namespace's reservations visible.

In an Interview

"A pod is stuck in Pending — how do you debug it?" is extremely common, and it's a good question because the answer is short if you know where to look.

What's being tested

• Do you know Pending is a scheduling problem? Not a kubelet, image, or application problem.
• Do you read the FailedScheduling message? It contains the whole answer.
• Do you know requests drive scheduling? The top-versus-describe distinction.

How to answer

Be direct about the first move: "Pending means it hasn't been scheduled, so kubectl describe pod and read the FailedScheduling event — it lists every node and which filter rejected it."

Then enumerate the usual causes: insufficient allocatable resource against requests, an untolerated taint, unsatisfiable node affinity or selector, an unbound PVC, and pod anti-affinity that can't be satisfied.

Then the subtlety worth volunteering: "One thing that catches people out is that kubectl top shows usage while the scheduler uses requests — a cluster can look idle and still be full. kubectl describe node under Allocated resources shows the scheduler's view."

Follow-ups to expect

"What if there are no events at all?" Events are garbage collected after roughly an hour, so an old Pending pod may have none left. Delete it and let the controller recreate it to get fresh events — and note that this is a strong argument for shipping events to a monitoring system.

"How do taints and tolerations differ from node affinity?" Opposite directions. A taint is the node repelling pods; affinity is the pod selecting nodes. A toleration permits but doesn't attract, so reserving nodes for a workload needs both.

"What if it's Pending because of a PVC?" kubectl describe pvc shows why it's unbound — a missing StorageClass, no default class, or an unsupported access mode. And with WaitForFirstConsumer, a Pending PVC is expected until a pod is scheduled, which isn't a fault at all.

Next Scenario

This pod never started. The last scenario involves pods that start perfectly, pass every check, and still drop user requests on every single deploy.