The Kubernetes Pod Lifecycle

Introduction¶

When you kubectl apply a Deployment, a chain of controllers and the kubelet turn that spec into running processes. Knowing the sequence tells you where a stuck pod is stuck.

Creation sequence¶

1. API server validates and stores the Pod object (created by the ReplicaSet controller from your Deployment).
2. Scheduler watches for pods with no nodeName, picks a node based on resource requests, affinity, taints/tolerations, and writes nodeName back.
3. kubelet on that node sees the pod, and: - pulls images (respecting imagePullPolicy) - runs init containers sequentially, each to completion - creates the pause/sandbox container (the shared network namespace) - starts app containers - runs postStart hooks, then startup → readiness → liveness probes
4. Pod becomes Ready when all containers are ready and any readiness gates pass; the endpoints controller adds it to Service endpoints.

Phase vs container state¶

Pod phase (.status.phase): Pending → Running → Succeeded/Failed (or Unknown). Coarse.

Container state (.status.containerStatuses[].state): Waiting (with a reason like ContainerCreating, ImagePullBackOff, CrashLoopBackOff), Running, or Terminated (with exit code and reason). This is where the useful detail is.

kubectl get pod mypod -o wide
kubectl describe pod mypod           # Events at the bottom = the story
kubectl get pod mypod -o jsonpath='{.status.containerStatuses[*].state}'

Init containers¶

Run in order, each must exit 0 before the next starts, all before app containers. Use for: waiting on a dependency, running migrations, fetching config, setting kernel params via a privileged init.

A failing init container puts the pod in Init:Error / Init:CrashLoopBackOff and the app never starts.

Probes¶

• startup — disables the other two until it passes once. For slow-booting apps; prevents liveness from killing them during startup.
• readiness — gates Service traffic. Failing readiness removes the pod from endpoints but does not restart it.
• liveness — failing it restarts the container.

Misconfigured liveness is a top cause of self-inflicted CrashLoopBackOff — see Debugging CrashLoopBackOff.

Termination¶

kubectl delete pod / a rollout / a scale-down triggers:

1. Pod marked Terminating, removed from Service endpoints immediately.
2. preStop hook runs (if defined).
3. SIGTERM sent to PID 1 in each container.
4. Kubernetes waits up to terminationGracePeriodSeconds (default 30).
5. SIGKILL if still running.

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 5"]   # let load balancers notice we're gone

The preStop: sleep 5 pattern buys time for the endpoint removal to propagate through kube-proxy / the ingress before your app stops accepting connections — avoids dropped requests during a rollout.

Your app must handle SIGTERM: stop accepting new work, finish in-flight requests, exit. If PID 1 is a shell script that doesn't forward signals, SIGTERM is swallowed and every shutdown takes the full grace period then SIGKILL. Use an init like tini, or exec the real process.

Verification and troubleshooting¶

kubectl describe pod mypod                     # Events
kubectl logs mypod -c app --previous           # logs from the crashed instance
kubectl get events --sort-by=.lastTimestamp -n myns

• Stuck Pending — scheduler can't place it: insufficient CPU/memory on any node, unsatisfied affinity, a taint with no toleration, or no PV available for a PVC. describe says which.
• Stuck ContainerCreating — image pull, a CNI failure, a failing volumeMount (secret/configmap doesn't exist), or the node's out of IPs.
• Init:0/2 — first init container hasn't completed. kubectl logs mypod -c <init-name>.
• Pod restarts cleanly but drops traffic during rollout — no preStop delay, or app doesn't drain on SIGTERM.
• Shutdown always takes 30s — SIGTERM not reaching/handled by PID 1.
• Terminating forever — a finalizer is stuck, or the node is gone. kubectl delete pod --grace-period=0 --force as a last resort.

Related tools and reading¶

• On-site: YAML validator, Base64 decoder.
• Related posts: Debugging CrashLoopBackOff, Liveness, readiness, and startup probes.