Linux Software RAID with mdadm

Introduction¶

mdadm manages Linux software RAID (the md driver). It's portable (move the disks to any Linux box), flexible (grow arrays, change levels), and free of proprietary hardware-controller lock-in. The trade-off vs hardware RAID is CPU for parity and no battery-backed write cache.

RAID levels¶

For 6+ large disks, prefer RAID 6 — with big drives, rebuild takes many hours and a second failure in that window is a real risk. RAID 5 on large drives is increasingly discouraged.

Create an array¶

mdadm --create /dev/md0 --level=6 --raid-devices=5 \
      /dev/sd[b-f]1
cat /proc/mdstat                       # watch the initial sync
mkfs.xfs /dev/md0
mount /dev/md0 /srv/data

Persist it:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf   # (or /etc/mdadm.conf on RHEL)
update-initramfs -u                               # Debian
dracut -f                                          # RHEL

The initramfs must know about the array or the root/boot won't assemble at boot. This step is skipped surprisingly often.

/etc/fstab by UUID (blkid /dev/md0), not /dev/md0 (the number can change).

Monitoring¶

cat /proc/mdstat
mdadm --detail /dev/md0

Set up email on failure — mdadm --monitor runs as a service:

# /etc/mdadm/mdadm.conf
MAILADDR [email protected]

systemctl enable --now mdmonitor        # RHEL: mdmonitor.service
mdadm --monitor --scan --test           # send a test alert

Also run periodic scrubs (read the whole array, fix mismatches) — most distros have a monthly cron/timer (/sys/block/md0/md/sync_action = check).

Replace a failed disk¶

mdadm --detail /dev/md0                 # find the failed device (state: faulty / removed)
mdadm /dev/md0 --fail /dev/sdc1         # mark it failed (if not already)
mdadm /dev/md0 --remove /dev/sdc1

# physically swap the disk, partition the new one identically:
sfdisk -d /dev/sdb | sfdisk /dev/sdc   # copy partition table from a good member

mdadm /dev/md0 --add /dev/sdc1         # rebuild starts automatically
watch cat /proc/mdstat                  # [====>....] recovery = X% (finish= Ymin)

Rebuild throttle: /proc/sys/dev/raid/speed_limit_{min,max} (KB/s) — raise for faster rebuild at the cost of IO for live workloads.

Growing¶

# add a disk and grow a RAID5/6
mdadm /dev/md0 --add /dev/sdg1
mdadm --grow /dev/md0 --raid-devices=6
# then grow the filesystem
xfs_growfs /srv/data          # or: resize2fs /dev/md0

Reshapes are slow and risky — back up first, and don't interrupt (a --backup-file on a separate device helps recovery).

The write hole¶

RAID ⅚: if power fails between writing data and its parity, the stripe is inconsistent and a subsequent disk failure can corrupt data silently. Hardware RAID uses battery-backed cache; mdadm's answer is a journal device (a small SSD/NVMe): mdadm --create ... --write-journal /dev/nvme0n1p1. Or use a filesystem with its own integrity (ZFS/btrfs RAID) for the important stuff.

Verification and troubleshooting¶

mdadm --detail /dev/md0
mdadm --examine /dev/sdb1               # per-disk superblock (array UUID, role, events)
smartctl -a /dev/sdb                    # is the disk actually failing?
journalctl -k | grep -i md

• Array not assembled at boot / boots to emergency mode — mdadm.conf not updated or not in the initramfs. Assemble manually (mdadm --assemble --scan), then fix the config + initramfs.
• mdadm: cannot open /dev/sdb1: Device or resource busy — the disk is already a member of an auto-assembled array, or held by LVM/a mount. wipefs -a a truly-spare disk before reuse.
• Rebuild extremely slow — speed_limit_max low, or the live workload is competing. Also a second marginal disk (SMART errors) slows every read.
• Array shows more events on some members than others after a crash — mdadm --assemble --force picks the most-consistent set; you may lose the last few writes. Investigate why (dirty shutdown, bad cable).
• "spare" disk not being used — it's a spare, not active; on a failure it auto-rebuilds. Force with --add after --removeing the failed one if it doesn't.
• Wrong disk pulled during replacement — pulling a good disk from a degraded RAID5 = data loss. Verify the serial (smartctl -i, ls -l /dev/disk/by-id/) before touching hardware; use disk activity LEDs (ledctl).
• Filesystem full but array grew — you grew the array but not the filesystem (xfs_growfs / resize2fs).

Related tools and reading¶

• On-site: Linux commands reference.
• Related posts: Choosing a Linux filesystem, LUKS full-disk encryption.