The Declarative Model and Reconciliation Loops

The Idea Underneath Everything

Most tools you've used are imperative: you issue commands, they execute, they report success or failure. docker run starts a container. If it fails, you get an error.

Kubernetes is declarative. You describe the state you want and store it in the cluster. Separate processes — controllers — continuously compare that description against reality and act to close the gap.

The consequence is the single most important thing to internalise early:

kubectl apply succeeding means the API server accepted and stored your object. It does not mean anything has happened yet.

Everything that follows in this roadmap — why pods sit Pending, why a Service silently drops traffic, why a config change appears to do nothing — is easier once you stop reading apply as "done".

The Control Loop

Every controller runs the same loop:

You declare replicas: 3. The ReplicaSet controller observes 2 running pods, notices the difference, creates one. Loop again: 3 and 3, nothing to do. A node dies, count drops to 2, the loop creates another.

Nobody told Kubernetes "a node died, please recover". It re-derives what to do from the current state, every time.

spec and status

Every object carries the same four sections:

apiVersion: apps/v1        # which API group and version
kind: Deployment           # what type of object
metadata:                  # name, namespace, labels, annotations
  name: api
spec:                      # WHAT YOU WANT — you write this
  replicas: 3
status:                    # WHAT IS — controllers write this
  readyReplicas: 2

You own spec. Controllers own status. When they disagree, something is either in progress or stuck — and telling those apart is most of Kubernetes debugging.

kubectl get deploy api -o jsonpath='{.spec.replicas}{"\n"}{.status.readyReplicas}{"\n"}'

Why a Perfect Manifest Can Do Nothing

Here's where the declarative model surprises people, and it's the failure mode worth building a habit around.

Your manifest can be syntactically valid, schema-correct, accepted by the API server — and still never produce a running pod. The API server validated the shape of your request. It did not promise the cluster can satisfy it.

Common cases:

• No node has enough capacity. Pod stays Pending.
• Every node carries a taint the pod doesn't tolerate. Pending.
• The PVC can't bind because no StorageClass matches. Pending.
• The image doesn't exist or registry credentials are wrong. ImagePullBackOff.
• A label selector doesn't match anything. Everything reports healthy and no traffic flows.

None of these produce an error at apply time. The object was fine. Reality just can't accommodate it.

Where the real answer lives

Controllers record what they tried and why it didn't work — as events, attached to the object:

kubectl describe pod api-7d4f8b9c5-x2kfp     # events at the bottom
kubectl get events --sort-by=.lastTimestamp  # cluster-wide, newest last
kubectl get events --field-selector involvedObject.name=api-7d4f8b9c5-x2kfp

kubectl describe is the single highest-value debugging command in Kubernetes, and the events section at the bottom is why.

Labels and Selectors: the Universal Join

Kubernetes objects don't reference each other by ID. They match on labels, and this loose coupling is used absolutely everywhere.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      app: api          # which pods this Deployment owns
  template:
    metadata:
      labels:
        app: api        # labels stamped onto created pods
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api            # which pods this Service sends traffic to

Three separate places, one label. The Deployment finds its pods by it, and the Service independently finds the same pods by it. Neither knows the other exists.

That decoupling is powerful and it has one sharp edge: a selector that matches nothing produces no error. A Service with a typo'd selector is a perfectly valid Service with zero endpoints. It accepts connections and drops them. Everything reports healthy.

kubectl get endpointslices -l kubernetes.io/service-name=api   # is anything behind the Service?
kubectl get pods -l app=api                                    # does the selector match pods?

Imperative and Declarative, Used Deliberately

Both styles work, and they have different jobs.

# Imperative — fast, unreproducible. For exploring.
kubectl create deployment api --image=myapp:1.0
kubectl scale deployment api --replicas=5
 
# Declarative — reproducible, reviewable, the source of truth.
kubectl apply -f deployment.yaml

The rule most teams settle on: imperative for investigation, declarative for anything that matters. An imperative change is invisible to your Git repository, so the next apply silently reverts it — and nobody can explain why production drifted back.

A genuinely useful hybrid is generating a manifest rather than hand-writing it:

kubectl create deployment api --image=myapp:1.0 --dry-run=client -o yaml > deployment.yaml

Namespaces

Namespaces partition a cluster into virtual clusters — names must be unique within one, not across. They're the scope for RBAC, resource quotas and network policy, which makes them the main multi-tenancy boundary.

kubectl get pods                      # current namespace only
kubectl get pods -n kube-system       # a specific one
kubectl get pods -A                   # all namespaces

What's Next

You understand how Kubernetes decides what should exist. The next guide covers the thing it actually creates — the pod — including init containers and the native sidecar support that became stable in v1.33.