Diagnosing Disk I/O with iostat

Introduction¶

When a box feels slow and CPU is idle, disk I/O is the usual suspect. iostat (from sysstat) shows per-device throughput and latency; iotop/pidstat show which process is responsible.

iostat -x¶

iostat -x 2 5        # extended stats, every 2s, 5 samples (ignore the first — it's since boot)

Device   r/s    w/s   rkB/s   wkB/s  rrqm/s wrqm/s  r_await w_await aqu-sz  %util
nvme0n1  120.0  450.0 15360   57600   0.0    30.0    0.35    1.20    0.68    42.0

%util is misleading on SSDs¶

On a single spinning disk, %util ~100% means saturated. On an SSD or a RAID array or SAN LUN that services many requests in parallel, %util can hit 100% while the device has tons of headroom — it just always has something queued. On modern storage, trust await and aqu-sz, not %util.

• High await + high aqu-sz + high throughput → genuinely saturated.
• High await + low aqu-sz + low throughput → the device is slow per-op (dying disk, degraded RAID, noisy-neighbour SAN, thermal throttling).
• Low await + high %util → busy but coping. Fine.

Finding the culprit¶

sudo iotop -oPa            # -o only active, -P processes, -a accumulated
pidstat -d 2               # per-process kB_rd/s, kB_wr/s, iodelay
sudo iotop -b -n 3 -o      # batch mode for logging

# Which files? 
sudo fatrace -f W          # writes, live
sudo lsof +D /var/lib/mysql

iodelay in pidstat -d is the time a process spent blocked on I/O — a high value points straight at the victim (which may not be the culprit).

Common bottlenecks¶

• Database with a small buffer pool / cache — constant random reads. Fix in the DB, not the disk (see InnoDB buffer pool sizing).
• fsync storms — every transaction/commit forcing a flush. w_await spikes correlate with commit rate. Battery-backed cache, or batch commits.
• Log-heavy app — synchronous logging to disk. Async logging, or a faster device for logs.
• Swap — si/so in vmstat non-zero; the disk is doing paging. Add RAM / fix the memory hog.
• Backup / cron job — a nightly rsync/mysqldump/updatedb saturating the disk. ionice -c3 (idle class) for those.
• Full or fragmented filesystem — > ~85% full slows allocation on some filesystems.
• Noisy neighbour (cloud/VM) — EBS gp3 baseline vs provisioned IOPS, or a shared SAN. await high with nothing local explaining it.

Verification and troubleshooting¶

vmstat 1 5                 # bi/bo blocks in/out, wa = CPU wait-for-IO %
iostat -x 1                # live
cat /sys/block/nvme0n1/queue/scheduler   # none/mq-deadline/bfq
smartctl -a /dev/sda | grep -Ei 'reallocated|pending|error'   # dying disk?

• wa (iowait) high but iostat shows the disk idle — waiting on a network filesystem (NFS/CIFS) or a remote block device, not local disk.
• await high only during a specific window — cron. Check /etc/cron*, systemctl list-timers.
• One process shows huge write in iotop but df doesn't grow — overwriting in place (a database, a circular log) or writing then deleting (temp files).
• RAID array slow — cat /proc/mdstat for a rebuild/resync in progress, or a failed member forcing degraded reads.
• ionice had no effect — only the bfq/cfq I/O scheduler honours ionice; none/mq-deadline (common default for NVMe) ignores it. Use cgroup io.max / systemd IOReadBandwidthMax instead.

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: Understanding Linux load average, Disk full but df says otherwise.