Packaging: Helm and Kustomize

The Problem With Raw YAML

You have a Deployment, a Service, a ConfigMap and an HTTPRoute. They need to exist in dev, staging and production, differing only in replica count, resource sizes, image tag and hostname.

Copying the set three times works for about a month. Then the copies drift: a probe fixed in staging but not production, a limit raised in one place only. Nobody knows which differences are deliberate.

Two tools solve this in opposite ways.

Kustomize: Patching, No Templates

Kustomize is built into kubectl (kubectl apply -k). It takes plain, valid manifests and layers patches over them.

base/
  kustomization.yaml
  deployment.yaml
  service.yaml
overlays/
  staging/
    kustomization.yaml
    replicas.yaml
  production/
    kustomization.yaml
    replicas.yaml
    resources.yaml

# base/kustomization.yaml
resources:
  - deployment.yaml
  - service.yaml

# overlays/production/kustomization.yaml
namespace: production
resources:
  - ../../base
patches:
  - path: replicas.yaml
  - path: resources.yaml
images:
  - name: myapp
    newTag: 1.4.0
commonLabels:
  environment: production

# overlays/production/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6

kubectl kustomize overlays/production      # render to stdout
kubectl apply -k overlays/production       # render and apply

The defining property: every file is a valid Kubernetes manifest. Your editor validates them, your IDE autocompletes them, and you can read any file and know what it means without mentally evaluating a template.

Two features worth knowing:

configMapGenerator:
  - name: app-config
    files: [config.yaml]

This generates a ConfigMap with a content hash appended to its name, and updates all references. Change the config and the Deployment's pod template changes — which triggers a rollout automatically. That's a clean structural answer to the "my ConfigMap change did nothing" problem from phase 2.

Helm: Templating and Releases

Helm treats a set of manifests as a package — a chart — with templating and release tracking.

mychart/
  Chart.yaml
  values.yaml
  templates/
    deployment.yaml
    service.yaml
    _helpers.tpl

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

helm install api ./mychart -f values-production.yaml
helm upgrade --install api ./mychart -f values-production.yaml
helm rollback api 3
helm list

Helm's genuine advantages:

• Release tracking. Helm knows what it installed, so helm rollback reverts a whole release — including deleting resources the new version removed, which kubectl apply won't do on its own.
• A package ecosystem. Most third-party software ships a chart. Installing Prometheus is one command.
• Rich parameterisation. Conditionals, loops, and hundreds of values for software that must serve many configurations.

The cost is that templates aren't valid YAML until rendered. A misplaced indent in a nindent produces errors far from the mistake, and your tooling can't help because the file isn't a manifest yet.

Choosing

They also combine, and this is common in practice: render a third-party Helm chart, then apply Kustomize overlays to it. You get the ecosystem without having to express every local adjustment through someone else's values schema.

# kustomization.yaml
helmCharts:
  - name: prometheus
    repo: https://prometheus-community.github.io/helm-charts
    version: 25.8.0
    valuesFile: values.yaml
patches:
  - path: custom-resources.yaml

Render Before You Apply

This is the most valuable habit in the guide, and it applies regardless of which tool you use.

Never apply a manifest you haven't seen rendered.

# Helm — what would actually be created?
helm template api ./mychart -f values-production.yaml
 
# Helm — validate against the live cluster without installing
helm install api ./mychart --dry-run --debug
 
# Kustomize — render the overlay
kubectl kustomize overlays/production
 
# What would change against what's running?
kubectl diff -k overlays/production
helm diff upgrade api ./mychart -f values-production.yaml   # helm-diff plugin
 
# Full server-side validation, including admission webhooks, without persisting
kubectl apply -k overlays/production --dry-run=server

The distinction between the dry-run modes is worth knowing:

• --dry-run=client — local syntax and schema checking only.
• --dry-run=server — sends it to the API server for full validation, admission control and defaulting, but doesn't persist. This catches what client-side cannot: a policy engine rejecting your pod, a Pod Security Admission violation, an invalid reference.

Why this matters more now

You will increasingly work with manifests generated by an AI assistant, adapted from a blog post, or inherited from a colleague. These are frequently good starting points and using them is sensible.

But recall the lesson from phase 1: kubectl apply succeeding means the API server stored your object, not that the cluster can satisfy it — or that it does what you intended. A generated chart can render valid YAML that mounts the wrong path, selects no pods, or grants a ClusterRoleBinding where a RoleBinding was meant.

helm template, kubectl kustomize, kubectl diff and --dry-run=server are how you look at the actual objects before they exist. It takes seconds and it checks the thing you care about rather than your reading of a template.

Where GitOps Fits

Both tools pair naturally with GitOps — Argo CD or Flux watching a Git repository and continuously reconciling the cluster to match it.

The appeal is that it extends the reconciliation model from phase 1 to deployment itself: Git holds desired state, a controller closes the gap, and manual changes made in the cluster are reverted automatically. Drift stops being possible, and the repository becomes an accurate record of what's deployed.

What's Next

The last guide of this phase builds the debugging method that ties the roadmap together — a fixed triage order, how to read the common failure states, and getting a shell into a distroless pod that doesn't have one.