Scenario: Three Replicas, One Disk

The Symptom

A team moves a document-processing service to Kubernetes. It writes uploaded files to a shared directory, so they provision a PVC and mount it into the Deployment.

kubectl apply -f app.yaml
# deployment.apps/processor created
# persistentvolumeclaim/uploads created

Clean apply, no errors. Then:

kubectl get pods -l app=processor
# NAME                         READY   STATUS              RESTARTS   AGE
# processor-5f9c8b7d4-2mkqp    1/1     Running             0          4m
# processor-5f9c8b7d4-7xnvw    0/1     ContainerCreating   0          4m
# processor-5f9c8b7d4-j3plq    0/1     ContainerCreating   0          4m

One pod runs. Two have been ContainerCreating for four minutes. No restarts, no crashes, no logs — the containers never started, so there's nothing to log.

The Investigation

ContainerCreating means the kubelet is trying to set the pod up — pulling an image, mounting volumes — and hasn't finished. The events say why:

kubectl describe pod processor-5f9c8b7d4-7xnvw

Events:
  Warning  FailedAttachVolume  attachdetach-controller
    Multi-Attach error for volume "pvc-8a3f2e1b" Volume is already used by pod(s) processor-5f9c8b7d4-2mkqp
  Warning  FailedMount         kubelet
    Unable to attach or mount volumes: unmounted volumes=[uploads], timed out waiting for the condition

Multi-Attach error. The volume is already attached elsewhere and cannot be attached again.

The manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: uploads
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: processor
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: processor
          volumeMounts:
            - name: uploads
              mountPath: /data/uploads
      volumes:
        - name: uploads
          persistentVolumeClaim:
            claimName: uploads

The Mechanism

Two facts collide.

1. ReadWriteOnce means once per node, not once per pod.

The name suggests one writer. It actually means the volume can be attached read-write by pods on a single node at a time. Pods co-located on that node can share it; pods on different nodes cannot.

2. A Deployment gives every replica the same PVC.

claimName: uploads is a single, named claim. All three replicas reference it. The scheduler spread them across three nodes — correct behaviour for availability — and now two of them need a volume that's attached to a third node's kubelet.

The Fix

Which fix depends on what the workload actually needs — worth establishing before reaching for a tool.

If each replica needs its own storage → StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: processor
spec:
  serviceName: processor-headless
  replicas: 3
  template:
    spec:
      containers:
        - name: processor
          volumeMounts:
            - name: uploads
              mountPath: /data/uploads
  volumeClaimTemplates:
    - metadata:
        name: uploads
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 50Gi

volumeClaimTemplates generates one PVC per replica — uploads-processor-0, uploads-processor-1, uploads-processor-2. Each pod gets its own volume, and because StatefulSet names are stable, each reattaches to its own data after rescheduling.

If replicas genuinely must share files → ReadWriteMany

spec:
  accessModes: [ReadWriteMany]
  storageClassName: efs-sc        # a filesystem, not block storage

RWX requires a backend that supports it — EFS, Filestore, Azure Files, NFS, CephFS. Most cloud block storage (EBS, GCE PD, Azure Disk) is RWO only, and asking for RWX on a block class leaves the PVC Pending indefinitely.

Often the best answer: don't use a shared filesystem

For uploaded files, object storage (S3, GCS) sidesteps the problem. No volumes, no access modes, no node affinity, and it scales independently of pod count. A shared POSIX filesystem across replicas is frequently a design choice worth revisiting rather than a constraint to engineer around.

Verifying

kubectl get pvc                       # one PVC per replica for a StatefulSet
kubectl get pods -l app=processor     # all replicas Running
kubectl describe pod <pod> | grep -A 5 Events   # no multi-attach warnings

In an Interview

Storage questions are common and often superficial. This scenario lets you show depth quickly.

What's being tested

• Do you know what RWO means? A specific fact most people get wrong.
• Do you know when a StatefulSet is the right tool? Beyond "it's for databases".
• Do you understand that apply succeeding isn't a guarantee? The core Kubernetes mental model.

How to answer

Read the error precisely: "A multi-attach error means the volume is already attached to another node. ReadWriteOnce is node-scoped, not pod-scoped, and a Deployment gives every replica the same PVC — so once the scheduler spreads replicas across nodes, only one can attach."

Then the options, with the reasoning: "If each replica needs its own storage, a StatefulSet with volumeClaimTemplates gives one PVC per replica. If they genuinely need to share, you need an RWX-capable backend — and most cloud block storage isn't. Often the better answer is object storage, because a shared POSIX filesystem across replicas is usually an assumption carried over from single-VM deployments."

Follow-ups to expect

"Why didn't this fail in staging?" Very likely a single-node cluster, where all replicas landed on the same node and RWO permitted it. Multi-node topology is what surfaces it — a good argument for staging that matches production's shape, not just its manifests.

"When would you use a StatefulSet over a Deployment?" When pods need stable identity, per-replica storage, or ordered startup. And worth adding: a StatefulSet gives you identity and storage, not replication, failover or backups — for a real database, an operator or managed service is usually the right call.

"How would you catch this before production?" kubectl describe on anything not in the expected state, as a deployment-verification step. Also --dry-run=server, though it won't catch this one — attachment is a runtime outcome, not an admission-time check. That distinction is itself worth saying.

Next Scenario

This failure had a clear error message once you ran describe. The next one has one too — and the message is a compressed summary of every filter the scheduler applied, which repays learning to read.