Scenario: The Error Nobody Could Reproduce

The Symptom

Intermittent database connection errors in production. Roughly one request in three.

Everything looks fine:

kubectl get pods -l app=api
# NAME                   READY   STATUS    RESTARTS   AGE
# api-7d4f8b9c5-x2kfp    1/1     Running   0          8d
# api-7d4f8b9c5-p9wzt    1/1     Running   0          8d
# api-7d4f8b9c5-k4mns    1/1     Running   0          2h

Three pods, same Deployment, same ReplicaSet hash, all Ready, no restarts. The application logs show connection failures — but only from some pods, and only sometimes.

Developers can't reproduce it locally. Retrying the request usually works. It gets labelled "flaky" and lives in the backlog for two weeks.

The Investigation

The breakthrough comes from comparing the pods rather than looking at them one at a time:

for p in $(kubectl get pods -l app=api -o name); do
  echo "=== $p"
  kubectl exec $p -- printenv DB_HOST
done

=== pod/api-7d4f8b9c5-x2kfp
postgres-primary.production.svc.cluster.local
=== pod/api-7d4f8b9c5-p9wzt
postgres-primary.production.svc.cluster.local
=== pod/api-7d4f8b9c5-k4mns
postgres-rw.production.svc.cluster.local

Same Deployment. Same image. Different configuration.

The ConfigMap had been updated 2 hours ago when the team migrated to a new database service name. The newest pod — created after the edit — picked up the new value. The two older pods are still using the old hostname, which no longer resolves.

One request in three fails because one pod in three is wrong.

The Mechanism

The ConfigMap was consumed as environment variables:

    envFrom:
      - configMapRef:
          name: app-config

Environment variables are injected once, at container start, and never change. There is no mechanism by which a running process's environment could update — the variable was set at exec time.

So editing the ConfigMap did exactly nothing to the running pods. It only affected pods created afterwards.

The third pod existed because something — a node drain, an eviction, a manual delete — had recreated it after the edit. Nobody connected the two events.

The Fix

Immediate

kubectl rollout restart deploy/api

This patches an annotation on the pod template, creating a new ReplicaSet and rolling all pods safely. Every pod now starts with current configuration.

Structural

Make config content part of the pod template, so a config change is a template change and triggers a rollout on its own.

With Kustomize, configMapGenerator appends a content hash to the ConfigMap's name and updates references:

configMapGenerator:
  - name: app-config
    files: [config.yaml]

Change the file and the generated name changes, the pod template changes, and a rollout happens automatically.

With Helm, the equivalent is a checksum annotation:

spec:
  template:
    metadata:
      annotations:
        checksum/config: "{{ include (print $.Template.BasePath \"/configmap.yaml\") . | sha256sum }}"

Verifying

# All pods should now report identical config
for p in $(kubectl get pods -l app=api -o name); do
  kubectl exec $p -- printenv DB_HOST
done | sort -u          # exactly one line = consistent

That sort -u producing a single line is the proof. It's also worth adding as a periodic check for anything config-sensitive.

In an Interview

"How do you manage configuration in Kubernetes?" sounds like a definitions question. Treating it as a failure-modes question is what makes an answer memorable.

What's being tested

• Do you know env vars don't update? The single most useful fact about ConfigMaps.
• Do you understand what triggers a rollout? Pod template changes — and nothing else.
• Can you debug non-deterministic failures? Comparing pods rather than inspecting one.

How to answer

Start with the mechanism: "ConfigMaps and Secrets decouple config from the image. The key thing to know is that values injected as environment variables are fixed at container start and never update, while mounted files do get refreshed — but only help if the application re-reads them."

Then the consequence: "So editing a ConfigMap appears to do nothing, and worse, it creates drift as pods get recreated for unrelated reasons. The fix is to make the config's content part of the pod template — a checksum annotation in Helm, or configMapGenerator in Kustomize — so a config change naturally triggers a rollout."

Follow-ups to expect

"How would you debug an error that only happens sometimes?" Compare instances rather than examining one. Intermittent plus environment-dependent usually means either a race or drift; comparing the actual resolved state across pods distinguishes them quickly.

"Are Secrets encrypted?" No — base64 encoded, which is an encoding not encryption. Real protection is encryption at rest in etcd (opt-in, configured by the operator), RBAC restricting who can read them, and keeping plaintext out of Git via External Secrets Operator or Sealed Secrets.

"How does GitOps help here?" Argo CD or Flux continuously reconciles the cluster against Git, so manual changes are reverted and the repository is an accurate record of what's deployed. Drift stops being something that can quietly accumulate.

Next Scenario

This incident was caused by config that didn't change. The next one is caused by something that changed far too fast — an autoscaler that responded to a traffic spike by taking down the database.