A node_exporter Metrics Guide

Introduction¶

node_exporter exposes Linux host metrics for Prometheus. It has a lot of them; this is the working subset with the queries and thresholds you actually use.

CPU¶

node_cpu_seconds_total is a counter per CPU per mode. Usage = everything that isn't idle:

# per-instance CPU utilisation (0-1)
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# iowait — high means disk-bound
avg by (instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m]))

# steal — the hypervisor took CPU from your VM (noisy neighbour)
avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m]))

Alert: sustained utilisation > 0.9 for 15m (warning), or iowait > 0.25, or steal > 0.1 (your VM is being starved).

Memory¶

Use MemAvailable, not MemFree. MemFree looks alarmingly low on a healthy box because Linux uses free RAM for cache.

# fraction of memory actually available
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# committed / swap
rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m])   # swap activity — should be 0

Alert: MemAvailable / MemTotal < 0.1 for 10m; any sustained swap-in/out.

Disk space and inodes¶

# space free, excluding tmpfs/overlay/etc.
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs|iso9660"}
  / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs|iso9660"}

# inodes free (df -i)
node_filesystem_files_free / node_filesystem_files

# predict full in 4 hours
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

# filesystem mounted read-only (a disk error remounted it ro)
node_filesystem_readonly == 1

Alert: < 10% space (warning) / < 5% (critical); < 5% inodes; predict_linear going negative within a few hours; node_filesystem_readonly == 1 (critical — a disk problem).

Disk I/O¶

# IOPS and throughput
rate(node_disk_reads_completed_total[5m])
rate(node_disk_read_bytes_total[5m])

# average latency per op (ms)
rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) * 1000

# device saturation (fraction of time busy) — see the iostat post for the SSD caveat
rate(node_disk_io_time_seconds_total[5m])

Network¶

rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*"}[5m]) * 8   # bits/s
rate(node_network_receive_errs_total[5m])          # errors — should be 0
rate(node_network_receive_drop_total[5m])          # drops — buffer pressure
node_network_up{device!~"lo"}                       # link state

Alert: any sustained errs/drop rate; node_network_up == 0 on a NIC you expect up.

Load and processes¶

node_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"})   # load per core
node_load5 / on(instance) group_left count by (instance) (node_cpu_seconds_total{mode="idle"})

# process/thread counts, fd usage
node_procs_running
node_filefd_allocated / node_filefd_maximum       # system-wide open FDs

Load-per-core > ~1.5 sustained warrants a look; see understanding load average for why the number alone doesn't tell you if it's a problem.

Time sync¶

node_timex_sync_status == 0        # NTP not synced
abs(node_timex_offset_seconds) > 0.1

Other high-value ones¶

• node_systemd_unit_state{state="failed"} == 1 — a failed systemd unit.
• node_boot_time_seconds — time() - node_boot_time_seconds < 600 = a host rebooted in the last 10 min (unexpected reboot alert).
• node_hwmon_temp_celsius — hardware temps (bare metal).
• node_edac_correctable_errors_total — ECC memory errors (a dying DIMM).
• node_textfile_scrape_error — a textfile-collector script is broken.

Verification and troubleshooting¶

curl -s localhost:9100/metrics | grep -E '^node_(cpu|memory|filesystem)' | head

• Metrics missing for a collector — it's disabled by default (--collector.X) or errored (node_scrape_collector_success{collector="X"} == 0).
• node_filesystem_* for weird mounts flooding dashboards — filter fstype!~"tmpfs|overlay|..." and mountpoint!~"/(run|sys|dev|var/lib/docker)...".
• CPU usage sums to > 100% — you forgot avg by (instance) (you're summing per-CPU rates). avg for a percentage, sum for "total cores used".
• MemAvailable missing — very old kernel; fall back to MemFree + Buffers + Cached (less accurate).
• Disk predict_linear false alarms — a backup/log burst skews the 6h trend; use a longer window or combine with an absolute threshold.
• High steal on a cloud VM — noisy neighbour / you're on a burstable instance out of credits. Resize or move.

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: PromQL for beginners, Diagnosing disk I/O with iostat.