Probes: Liveness, Readiness and Startup

Three Probes, Three Questions

Probes are where a large share of production incidents are created — not because they're complex, but because the three are easy to conflate. Each answers a different question and each has a different consequence when it fails.

The distinction to hold onto: readiness removes traffic, liveness kills the container. Choosing the wrong one turns a recoverable blip into an outage.

Readiness: Controlling Traffic

A readiness probe decides whether a pod appears in its Service's endpoints. Fail it and traffic stops arriving — but the container keeps running and can recover.

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

This is the right response to temporary unavailability: a full connection pool, a warming cache, a dependency that's briefly unreachable. The pod steps out of rotation, recovers, and steps back in — with no restart and no lost in-memory state.

Readiness is also what makes rolling updates safe. From the previous guide: without a readiness probe, a pod counts as available the moment it starts, so Kubernetes retires an old pod and replaces it with one that can't serve yet. Readiness is the difference between a zero-downtime rollout and a rollout that drops requests.

Liveness: the One That Can Hurt You

A liveness probe restarts the container when it fails. That makes it useful for exactly one thing: a process that is alive but permanently stuck — a deadlock, an event loop wedged forever.

It is dangerous precisely because restarting is a big hammer.

livenessProbe:
  httpGet:
    path: /healthz      # must NOT check external dependencies
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

A liveness probe should answer only: is this process still capable of making progress on its own? Usually that means a trivial endpoint that returns 200 if the event loop is responsive, touching nothing external.

Startup: Protecting Slow Boots

The problem startup probes solve: a JVM or heavyweight framework takes 60 seconds to boot. Your liveness probe checks every 10 seconds with a threshold of 3 — so at 30 seconds it declares the container dead and restarts it. It never finishes starting. You get CrashLoopBackOff for an application that has nothing wrong with it.

The old workaround was a large initialDelaySeconds on the liveness probe — which forces you to trade slow failure detection for slow startup tolerance.

A startup probe separates the two concerns:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 30        # allows up to 150s to boot
 
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3         # once started, detects a hang in ~30s

While the startup probe is running, the liveness probe is disabled entirely. Once it passes once, liveness takes over with its tight timings. Slow boot tolerated, fast failure detection retained.

Probe Types and Timing

Four handlers are available:

httpGet:    { path: /ready, port: 8080 }      # 2xx or 3xx = success
tcpSocket:  { port: 5432 }                     # can the port be connected to?
exec:       { command: ["sh","-c","pg_isready"] }  # exit 0 = success
grpc:       { port: 9090 }                     # gRPC health checking protocol

tcpSocket is weak — a port being open says little about whether the application works. Prefer httpGet or grpc where you can.

The timing fields combine into a real budget worth calculating:

Detection time is roughly periodSeconds × failureThreshold. With defaults that's ~30 seconds before a liveness restart.

Reading Probe Failures

kubectl describe pod api-7d4f8b9c5-x2kfp

The events show exactly which probe failed and why:

Warning  Unhealthy  2m (x3 over 2m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
Warning  Unhealthy  1m (x3 over 1m)  kubelet  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": context deadline exceeded
Normal   Killing    1m               kubelet  Container api failed liveness probe, will be restarted

That third line is the one to look for when a pod is restarting and you can't work out why. context deadline exceeded means the probe timed out — often a timeoutSeconds that's too tight rather than a genuinely broken application.

A Sensible Default Configuration

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30          # generous boot budget
 
readinessProbe:
  httpGet: { path: /ready, port: 8080 }   # checks dependencies
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3
 
livenessProbe:
  httpGet: { path: /healthz, port: 8080 } # checks NOTHING external
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

Two endpoints doing two different jobs: /healthz proves the process is responsive, /ready proves it can actually serve a request including its dependencies.

What's Next

Your workloads run and receive traffic at the right moment. The next guide covers their configuration — and the ConfigMap behaviour that makes a config change appear to do absolutely nothing.