Liveness, Readiness, and Startup Probes

Introduction¶

Kubernetes probes let the kubelet check container health. There are three, they do different things, and mixing them up is one of the most common self-inflicted outages.

The three probes¶

While a startup probe is defined and not yet succeeding, liveness and readiness are disabled.

Parameters¶

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 0      # prefer a startupProbe over a big initialDelay
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3         # 3 consecutive fails -> action
  successThreshold: 1
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30        # 30 * 5s = 150s to start before giving up

Probe handlers: httpGet (2xx/3xx = pass), tcpSocket (connect = pass), exec (exit 0 = pass), grpc (1.27+).

Design patterns¶

Separate the endpoints¶

• /healthz (liveness) — cheap, no dependencies. Just "is the process responsive?". Return 200 unless the process is genuinely unrecoverable.
• /ready (readiness) — check dependencies you need to serve: DB pool alive, cache reachable, warmup done. Return 503 when you can't serve.

Do not check the database in the liveness probe. If the DB blips, every pod fails liveness, every pod restarts, and now you have a thundering herd against a recovering database. The DB belongs in readiness (stop taking traffic) not liveness (restart).

Slow starters¶

App takes 90s to load a model / warm a cache? Use a startupProbe with a generous failureThreshold * periodSeconds budget instead of initialDelaySeconds: 120 on liveness (which would also delay detecting a real hang later).

Graceful shutdown interaction¶

During termination, readiness should fail first (or the pod is removed from endpoints by the control plane anyway), then a preStop sleep, then SIGTERM. See The Kubernetes pod lifecycle.

Verification and troubleshooting¶

kubectl describe pod mypod | grep -A5 -iE 'liveness|readiness|startup'
kubectl get events --field-selector involvedObject.name=mypod | grep -i probe
kubectl exec mypod -- curl -sv localhost:8080/healthz

• CrashLoopBackOff, exit 137, no OOMKilled — liveness probe failing. describe shows "Liveness probe failed: ...". Causes: wrong path/port, timeoutSeconds: 1 too tight under load, probe hits an endpoint that checks a slow dependency, or initialDelaySeconds shorter than startup time (use a startup probe).
• Pod Running but gets no traffic — readiness failing. kubectl get endpoints <svc> won't list it. Check /ready returns 200.
• Rollout drops requests — readiness passes too eagerly (before the app can actually serve), or no preStop delay on shutdown.
• Probe works with curl from inside but fails as a probe — the kubelet probes the pod IP, not localhost; if the app binds 127.0.0.1 the probe can't reach it. Also check NetworkPolicy isn't blocking the kubelet.
• exec probe slow / zombie processes — each exec forks; a probe that spawns a shell every 5s on thousands of pods is load. Prefer httpGet.
• gRPC service: probe always fails — needs grpc probe type (1.27+) or a gRPC health-checking sidecar/binary; plain tcpSocket only checks the port is open.

Related tools and reading¶

• On-site: Port Checker, HTTP Header Analyzer.
• Related posts: Debugging CrashLoopBackOff, The Kubernetes pod lifecycle.