Debugging CrashLoopBackOff

Introduction¶

CrashLoopBackOff means a container keeps exiting and Kubernetes keeps restarting it with exponential backoff (10s, 20s, 40s… capped at 5 min). The status tells you nothing about why. Here's the systematic path.

Step 1: exit code and reason¶

kubectl describe pod mypod | grep -A5 'Last State'
kubectl get pod mypod -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

Step 2: the previous container's logs¶

The current container may not have started yet. You need the one that died:

kubectl logs mypod --previous
kubectl logs mypod -c <container> --previous     # multi-container pod

90% of "Error" crashes are explained here: an unset env var, a config file it can't find, a DB it can't reach, a port already in use, an unhandled exception on startup.

Step 3: match the cause¶

App crashes immediately¶

Logs show a stack trace or "missing X". Fix the config: a ConfigMap/Secret key, an env var, a mounted file. Reproduce locally with the same env.

OOMKilled¶

kubectl describe pod mypod | grep -i oom
kubectl top pod mypod

The limit is too low, or there's a leak. Raise resources.limits.memory, or fix the app. Note: JVM/Node need -Xmx / --max-old-space-size set below the limit, or they'll grow into the OOM. See Requests vs limits and OOMKilled.

Failing liveness probe (exit 137, no OOM)¶

kubectl describe pod mypod | grep -A3 Liveness

The probe path/port is wrong, the initialDelaySeconds is too short for a slow boot (add a startup probe), or the app is genuinely wedged. Temporarily remove the liveness probe to confirm — if it then runs fine but is slow to be ready, it's a probe tuning problem.

Command / entrypoint wrong (127, 126)¶

command: overrides the image ENTRYPOINT; a typo → 127. A non-executable script or read-only root filesystem → 126. Check securityContext.readOnlyRootFilesystem.

Works then crashes after N seconds¶

Not a startup problem. Something at runtime: a scheduled task, a memory climb, a dependency timeout, a cert expiry check. Correlate the crash interval with app behaviour.

Step 4: get a shell without the crash¶

# Ephemeral debug container sharing the pod's namespaces (k8s 1.25+)
kubectl debug -it mypod --image=busybox --target=app

# Or run the image with the entrypoint overridden
kubectl run tmp --rm -it --image=myapp:latest --command -- sh

Verification and troubleshooting¶

kubectl get pod mypod -w                        # watch restart count climb
kubectl get events --field-selector involvedObject.name=mypod --sort-by=.lastTimestamp

• --previous says "not found" — the container never successfully started (still Waiting). The problem is image pull, a mount, or an init container, not the app. describe the pod.
• Logs are empty — app writes to a file not stdout, or crashes before logging. Override the entrypoint and run it by hand.
• Restart count high but pod now Running — it recovered (a dependency came up). Check whether the backoff just hadn't cleared; if truly stable, fix ordering with an init container or readiness on the dependency.
• Only crashes in the cluster, not locally — env difference: resource limits, a NetworkPolicy blocking egress, a read-only FS, a missing ServiceAccount token, DNS.
• CrashLoop right after a deploy — bad image tag, a changed env/secret, or a migration that didn't run. kubectl rollout undo deployment/myapp.

Related tools and reading¶

• On-site: Base64 decoder, YAML validator.
• Related posts: The Kubernetes pod lifecycle, Requests vs limits and OOMKilled.