kdump and Kernel Crash Analysis

Introduction¶

A kernel panic normally leaves you with a hung console and no evidence. kdump reserves a slice of RAM at boot for a minimal "capture kernel"; on a panic, the crashing kernel kexecs into it, and the capture kernel dumps the original kernel's memory (vmcore) to disk before rebooting. That vmcore + symbols lets you find the cause.

How it works¶

1. At boot, crashkernel= on the kernel cmdline reserves memory (e.g. 256M–512M) that the running kernel won't use.
2. kdump.service loads the capture kernel + initramfs into that reserved region via kexec.
3. On panic, the kernel kexecs directly into the capture kernel (skipping firmware/BIOS) which runs in the reserved memory.
4. The capture kernel mounts a target, runs makedumpfile to write a (filtered, compressed) vmcore, then reboots normally.

Configure (RHEL/Fedora)¶

# 1. reserve memory — grubby edits the cmdline
grubby --update-kernel=ALL --args="crashkernel=512M"
# (RHEL 9+ often auto-sizes with crashkernel=auto or a boot-time default)
reboot

# 2. check the reservation took
cat /proc/cmdline
kdumpctl showmem            # or: grep -c . /sys/kernel/kexec_crash_size   (nonzero)

# 3. configure the target: /etc/kdump.conf
#   local disk:
path /var/crash
core_collector makedumpfile -l --message-level 7 -d 31
#   or NFS / SSH:
# nfs my-nfs:/export/crash
# ssh kdump@dumpserver
# sshkey /root/.ssh/kdump_id_rsa

# 4. enable
systemctl enable --now kdump
kdumpctl status

Debian/Ubuntu: apt install kdump-tools, set USE_KDUMP=1 in /etc/default/kdump-tools, crashkernel= via /etc/default/grub.

makedumpfile -d 31 filters out zero/free/cache pages — a 128GB box's dump shrinks to a few GB. -d 0 keeps everything (rarely needed).

Test it (safely, on a box you can reboot)¶

# forces a panic RIGHT NOW — the machine will crash and capture
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

It panics, the capture kernel runs, writes to /var/crash/<timestamp>/vmcore, and reboots. If you don't get a vmcore, kdump isn't working — fix it before you need it.

First look with crash¶

dnf install crash kernel-debuginfo    # debuginfo must match the crashed kernel EXACTLY
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/127.0.0.1-*/vmcore

In the crash> prompt:

bt              # backtrace of the panicking task — the top frames are the cause
bt -a           # backtrace of every CPU (find the one that's not idle)
log             # the kernel ring buffer at crash time (the panic message + oops)
kmem -i         # memory usage — was it OOM?
ps              # process list at crash
dev -d          # disk I/O in flight
sys             # system summary: panic string, kernel version, load
foreach bt      # every task's stack

Start with sys (what was the panic string?) and bt (where). Common signatures: NULL pointer dereference in a specific driver, soft lockup / hung task (a CPU or task stuck — often I/O or a lock), general protection fault, out of memory.

Verification and troubleshooting¶

kdumpctl status
cat /sys/kernel/kexec_crash_loaded        # 1 = capture kernel is loaded and ready
journalctl -u kdump
ls -la /var/crash/

• kexec_crash_loaded is 0 — the capture kernel didn't load. Usually the crashkernel= reservation is too small (bump to 512M–1G, more on big/complex systems), the reservation failed (memory fragmentation — try crashkernel=X,high), or kdump.service errored.
• Panic happens but no vmcore — the target is unreachable from the minimal capture environment (a network dump needs the NIC driver in the capture initramfs; a local disk needs its driver + the filesystem). kdumpctl rebuild and check /etc/kdump.conf dracut_args includes the needed modules.
• crash fails "vmlinux and vmcore do not match" — debuginfo version mismatch. It must be the exact kernel that crashed (crash> won't start otherwise). Keep debuginfo for kernels you run.
• Dump is huge / fills the disk — -d 31 filtering not set, or -d 0. Also ensure /var/crash (or the remote target) has room for a few GB.
• Reserved memory "wasted" on a stable box — it is; ~512M off a server is usually acceptable insurance. Skip kdump only on tiny memory-constrained systems.
• Panic on a VM — the hypervisor may have its own crash-dump; kdump inside the guest still works and is often more useful. On some clouds you can't set the cmdline easily — check the platform docs.

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: The Linux OOM killer, explained, strace for troubleshooting.