ConfigMaps and Secrets: Why Your Config Change Did Nothing

The Goal: One Image, Every Environment

The same container image should run in dev, staging and production, with only its configuration differing. Bake config into the image and you need a rebuild per environment — which destroys the value of promoting a tested artifact.

Kubernetes provides two objects for this. They're nearly identical in mechanics and very different in what they promise.

• ConfigMap — non-sensitive configuration.
• Secret — sensitive values, with some additional handling.

Two Ways to Consume Them, One Crucial Difference

This is the most important thing in the guide, and it explains an enormous amount of confusion.

As environment variables

spec:
  containers:
    - name: api
      envFrom:
        - configMapRef: { name: app-config }
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef: { name: db-creds, key: password }

Simple, and every language reads env vars natively.

But environment variables are injected once, at container start, and never change. Update the ConfigMap and the running pod keeps the old value — forever. There is no mechanism by which it could update; the variable was set in the process's environment at exec time.

As mounted files

spec:
  containers:
    - name: api
      volumeMounts:
        - name: config
          mountPath: /etc/app
          readOnly: true
  volumes:
    - name: config
      configMap: { name: app-config }

Each key becomes a file. Mounted ConfigMaps and Secrets are updated by the kubelet after a change — eventually, subject to a sync delay.

But that only helps if your application re-reads the file. Most applications read config once at startup and never look again, so the file updates underneath a process that isn't watching.

Making a Config Change Actually Take Effect

Since the pods won't pick it up on their own, restart them:

kubectl rollout restart deploy/api

This patches an annotation on the pod template, producing a new ReplicaSet and a safe rolling replacement — the mechanism from the rollouts guide, used deliberately.

For an automatic version, put a hash of the config into the pod template so a config change is a template change:

spec:
  template:
    metadata:
      annotations:
        checksum/config: "{{ include (print $.Template.BasePath \"/configmap.yaml\") . | sha256sum }}"

That's the Helm idiom. The principle is general: make the config's content part of the pod template, so changing it naturally triggers a rollout. Without this, a ConfigMap edit is silent — and silently inconsistent, because pods created after the edit get the new value while older ones keep the old one. That's the configuration drift scenario from the roadmap: two pods, same Deployment, different behaviour, intermittent errors nobody can reproduce.

Secrets Are Not Encrypted

Be precise about this, because the misconception is widespread and consequential.

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
data:
  password: c3VwZXJzZWNyZXQ=      # base64

Base64 is an encoding, not encryption. Anyone who can read the Secret can decode it instantly:

kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d

There's no key, no algorithm, no protection. Base64 exists so binary data can travel in JSON — nothing more.

So what does a Secret give you beyond a ConfigMap?

• Values are not displayed by default in kubectl get / describe output.
• They can be encrypted at rest in etcd — but this must be explicitly configured; it is not on by default.
• They're a distinct resource type, so RBAC can control them separately from ConfigMaps.
• The kubelet only distributes a Secret to nodes running a pod that needs it.

Real protection comes from three things, none automatic:

1. Encryption at rest in etcd, configured by the cluster operator.
2. RBAC restricting who can read Secrets — the most commonly neglected step, since a get secrets permission is effectively a plaintext credential dump.
3. Keeping plaintext out of Git, which is where the ecosystem tools come in.

Keeping Secrets Out of Git

Two established approaches:

External Secrets Operator — Secrets live in a real secret manager (Vault, AWS Secrets Manager, GCP Secret Manager). You commit an ExternalSecret that references them, and the operator syncs the actual values into the cluster. Your repository contains pointers, never values, and you inherit the secret manager's access control, audit trail and rotation.

Sealed Secrets — encrypt a Secret with a cluster-held public key, producing a SealedSecret that is safe to commit. A controller in the cluster decrypts it. Only that cluster can, so committing it is safe.

The properties worth naming when choosing: access control (who may read this), audit (who did), and rotation (change it without redeploying everything). A committed Secret manifest offers none of the three.

Practical Notes

# Create from literals or files rather than hand-writing base64
kubectl create secret generic db-creds --from-literal=password='s3cr3t'
kubectl create configmap app-config --from-file=./config/
 
# Inspect
kubectl get configmap app-config -o yaml
kubectl describe secret db-creds        # shows keys and sizes, not values

A few behaviours worth knowing:

• ConfigMaps and Secrets are namespaced. A pod can only reference one in its own namespace — no cross-namespace references.
• There's a 1 MiB size limit, because they're stored in etcd. Large files belong on a volume or in object storage.
• A pod referencing a missing ConfigMap or Secret will not start. kubectl describe pod reports it clearly as a CreateContainerConfigError.
• immutable: true prevents changes to a ConfigMap or Secret and measurably reduces API server load at scale. You replace rather than edit — which pairs naturally with the rollout-on-change pattern.

What's Next

Your workloads run, stay healthy, and are configured. Phase 3 covers how they find each other and how traffic gets in from outside — including the part of the ecosystem that changed most recently, where a great deal of existing material will now actively mislead you.