Kubernetes Requests vs Limits, and OOMKilled

Introduction¶

Every container can declare resources.requests and resources.limits for CPU and memory. They do very different things, and the difference explains most "why did my pod get killed" and "why won't it schedule" questions.

Requests: scheduling and QoS¶

requests is what the scheduler reserves. A node with 4 CPU and 8 Gi advertises that as allocatable; the scheduler places pods until the sum of requests fills it. Requests are not enforced at runtime — a container can use more than it requested if the node has spare capacity.

Requests also set the pod's QoS class, which decides eviction order under node pressure:

Limits: enforcement¶

• CPU limit — enforced by CFS throttling. Exceed it and the kernel throttles the container (pauses it until the next scheduling period). The pod keeps running, just slower. Aggressive CPU limits cause mysterious latency spikes.
• Memory limit — a hard wall. Exceed it and the kernel OOM-kills the container (exit 137, reason OOMKilled). It's then restarted per the pod's restart policy.

resources:
  requests:
    cpu: "250m"        # 0.25 of a core
    memory: "256Mi"
  limits:
    cpu: "1"           # can burst to 1 core, then throttled
    memory: "512Mi"    # OOMKilled above this

Why JVMs and Node get OOMKilled¶

The runtime doesn't know about the cgroup limit unless told, or it does but allocates non-heap memory (metaspace, thread stacks, direct buffers, JIT code) on top of the heap.

• Java: modern JVMs are container-aware (-XX:+UseContainerSupport, default on). Set -XX:MaxRAMPercentage=75 rather than a fixed -Xmx, and leave headroom — the container limit must exceed heap + metaspace + stacks + native. A 512Mi limit with -Xmx512m will be OOMKilled.
• Node.js: default old-space is ~2 Gi regardless of the limit. Set --max-old-space-size=<~75% of limit in MB>.
• Python: no VM heap, but multiprocessing, ML libs, and glibc arena fragmentation add up. MALLOC_ARENA_MAX=2 helps.

Sizing method¶

1. Run with generous limits and no requests in staging under realistic load.
2. Observe actual usage: kubectl top pod, or a VerticalPodAutoscaler in recommender mode, or Prometheus container_memory_working_set_bytes / rate(container_cpu_usage_seconds_total[5m]).
3. Set memory request = limit near the observed peak + ~20% (memory doesn't compress; over-committing it causes node OOM).
4. Set CPU request to the typical usage; set the CPU limit high or omit it (throttling steady-state work is usually worse than the noisy-neighbour risk). This is debated — for latency-sensitive services, no CPU limit is a common choice.

Verification and troubleshooting¶

kubectl top pod mypod --containers
kubectl describe pod mypod | grep -A2 -i 'last state\|reason'
kubectl get pod mypod -o jsonpath='{.status.qosClass}'
# CPU throttling
kubectl exec mypod -- cat /sys/fs/cgroup/cpu.stat        # nr_throttled, throttled_usec

• OOMKilled right at startup — limit below the runtime's baseline. Raise it or cap the heap.
• OOMKilled under load only — request buffers / caches / connection pools scaling with traffic. Profile, then size for peak.
• Pod won't schedule ("Insufficient memory") — sum of requests exceeds node allocatable. Lower requests, add nodes, or check for a LimitRange inflating them.
• Latency spikes correlate with traffic, CPU usage below limit — CFS throttling. nr_throttled climbing confirms it. Raise or remove the CPU limit.
• One namespace's pods all Burstable when you set requests==limits — a LimitRange default is adding a different limit. kubectl get limitrange -n ns -o yaml.
• Node NotReady / pods evicted — node memory pressure from BestEffort or under-requested pods. Set requests everywhere.

Related tools and reading¶

• On-site: YAML validator.
• Related posts: Debugging CrashLoopBackOff, Node pressure and evictions.