Understanding Linux Load Average

Introduction¶

"Load is at 20!" is one of the most misread metrics in operations. On Linux specifically, load average counts something broader than CPU demand, and a "high" number can be completely benign or a real emergency depending on why.

What it measures¶

uptime / top show three numbers — the 1, 5, and 15-minute exponentially weighted averages of the number of tasks that are:

• Runnable (R) — running or waiting for a CPU, plus
• Uninterruptible (D) — blocked in the kernel, almost always on disk or network I/O.

That second part is Linux-specific (other Unixes count only runnable). It's why a box hammering a slow disk shows high load even with idle CPUs.

Reading the number¶

Compare load to the CPU count:

nproc                    # logical CPUs
uptime

• Load ≈ CPU count → the machine is fully utilised, no queue. Fine.
• Load ≫ CPU count and rising → tasks are queuing; things are getting slower.
• Load ≫ CPU count but CPUs are idle → the queue is D-state tasks waiting on I/O, not CPU. Adding CPUs won't help.

The ⅕/15 spread tells you the trend: 15.0 2.0 0.5 was a spike that's over; 0.5 2.0 8.0 is a problem that's building.

Find the real bottleneck¶

Is it CPU or I/O?¶

# %iowait vs %user/%sys; run-queue (r) vs blocked (b)
vmstat 1 5

# Per-device: %util near 100 and high await = disk is the limit
iostat -xz 1 3

# Pressure Stall Information (kernel 4.20+) — the cleanest signal
cat /proc/pressure/cpu
cat /proc/pressure/io

some avg10 in /proc/pressure/io above ~20 means tasks are stalling on I/O a fifth of the time — that's your answer.

Which processes?¶

# Sort by CPU
top -o %CPU

# Tasks currently in D state (uninterruptible I/O)
ps -eo state,pid,comm | awk '$1 ~ /D/'
for i in 1 2 3; do ps -eo state,pid,comm | grep '^D'; sleep 1; done

# What is a stuck D-state process waiting on?
cat /proc/<pid>/stack
cat /proc/<pid>/wchan

CPU-bound specifics¶

mpstat -P ALL 1        # is one core pinned while others idle? (single-threaded)
pidstat -t 1           # per-thread CPU

Verification and troubleshooting¶

• Load 8 on 8 cores, everything responsive — that's healthy full utilisation. Not an incident.
• Load 50, CPU 5% idle... no wait, CPU is 90% idle — I/O wait. iostat -xz; find the device at %util 100. Often a failing disk, a saturated network share, or a runaway fsync loop.
• Load high, one core at 100%, rest idle — a single-threaded hot process. mpstat -P ALL 1 confirms; the fix is in the app, not the box.
• Load spikes exactly on a schedule — a cron job, updatedb, logrotate, a backup. systemctl list-timers, journalctl --since around the spike.
• Load climbing with no obvious process — thousands of short-lived processes (a fork bomb, a broken shell loop, a CI runner). vmstat 1 — a huge r with low per-process CPU. dmesg for OOM kills.
• NFS/CIFS mount hung → load 200 — every process touching the mount goes D and never returns. mount | grep nfs, check the server, consider soft/intr mount options next time.

Related tools and reading¶

• On-site: SAR Report Analyzer for historical CPU/mem/load/net charts, Linux commands reference.
• Related posts: Disk full but df says otherwise, strace for troubleshooting.