Node Pressure and Pod Evictions

Introduction¶

A Kubernetes node has finite memory, disk, and process slots. When one gets scarce, the kubelet proactively evicts pods to reclaim it before the whole node falls over. Understanding the mechanism lets you protect what matters.

The signals¶

The kubelet watches:

Soft thresholds add an eviction-soft-grace-period (gentle, respects pod grace period). Hard thresholds evict immediately (grace period 0). Config is in the kubelet (--eviction-hard, --eviction-soft, or the KubeletConfiguration).

Eviction order¶

Under memory pressure, the kubelet ranks pods by:

1. Whether the pod exceeds its memory requests — pods using more than they requested go first.
2. Pod Priority (priorityClassName) — lower priority first.
3. How much the pod exceeds requests — biggest offender first.

Practical consequence of QoS:

• BestEffort (no requests/limits) — evicted first, always over "request".
• Burstable — evicted when using more than requested.
• Guaranteed (requests == limits) — evicted last, only under extreme pressure or if exceeding limits (which for Guaranteed means OOMKill anyway).

Disk pressure is different: it first deletes dead containers and unused images, then evicts pods (ranked by local disk usage vs requests, then priority).

Eviction vs OOMKill¶

• Eviction — the kubelet removes the pod gracefully-ish; the pod object gets status.reason: Evicted and stays around (a tombstone) until cleaned up. A controller reschedules it elsewhere.
• OOMKill — the Linux kernel kills a process because a cgroup hit its memory limit. Just that container, restarted in place.

You can hit both: node memory pressure evicts pods; a single pod exceeding its limit gets OOMKilled without any node pressure.

Preventing the wrong evictions¶

• Set requests on everything. A pod with no memory request is BestEffort and first to die, even if it's important.
• Set memory request == limit for critical workloads (Guaranteed QoS).
• PriorityClasses — give system/critical workloads a high priorityClassName; system-cluster-critical / system-node-critical exist for add-ons.
• Reserve resources for the system: --system-reserved / --kube-reserved so the kubelet's math leaves headroom for the OS and container runtime.
• Pod Disruption Budgets don't stop node-pressure evictions (those are involuntary) but do protect against voluntary disruptions like drains.
• Right-size so nodes aren't chronically near the threshold.

Verification and troubleshooting¶

kubectl describe node node-1 | grep -A10 Conditions   # MemoryPressure/DiskPressure/PIDPressure
kubectl get events -A --field-selector reason=Evicted
kubectl get pods -A --field-selector status.phase=Failed | grep Evicted
kubectl describe pod evicted-pod | grep -A3 Status
journalctl -u kubelet | grep -i evict            # on the node
df -h /var/lib/containerd /var/lib/kubelet        # disk pressure source

• Pods repeatedly evicted and rescheduled onto the same node — the node is chronically over-committed; the replacement lands there and gets evicted too. Fix requests, cordon and investigate, or add capacity.
• DiskPressure with plenty of df space — inodes (df -i), or it's imagefs (a separate filesystem for images) that's full, not nodefs.
• Lots of Evicted tombstone pods piling up — kubelet doesn't garbage collect them by default below a threshold. kubectl delete pod --field-selector status.phase=Failed -A, or set --pod-eviction-timeout / a cleanup job.
• Critical add-on (CNI, CSI) evicted — it lacks a system-node-critical priority class. Add one.
• Node flaps NotReady under load — pressure so severe the kubelet itself is starved; increase --kube-reserved/--system-reserved and lower the max pods per node.
• Eviction storm across the cluster — a memory leak in a DaemonSet hits every node at once. Roll it back; the fix isn't in eviction config.

Related tools and reading¶

• On-site: YAML validator.
• Related posts: Requests vs limits and OOMKilled, Disk full but df says otherwise.