Disk Full but df Says There's Space

Introduction¶

No space left on device, but df -h shows 30% free. Or du totals 40 GB on a 100 GB disk that's full. Both are real situations with distinct causes. Here's how to tell which one you have.

Cause 1: deleted files still held open¶

A process opened a log file, the file was rm'd (or logrotate rotated it without signalling the process), but the process still has the descriptor — so the space isn't freed until the process closes it or exits.

df counts it as used; du doesn't see it (no directory entry).

# Files that are deleted but still open, biggest first
sudo lsof -nP +L1 | awk '$5=="REG" {print $7, $1, $2, $9}' | sort -rn | head

# or
sudo lsof -nP | grep '(deleted)'

Fix: restart or HUP the offending process. For a one-off reclaim without a restart, truncate the file through its descriptor:

sudo truncate -s 0 "/proc/<pid>/fd/<fd-number>"

The classic offender is a service still writing to /var/log/app.log (deleted) after logrotate moved it — fix the logrotate config to copytruncate or send the right signal in postrotate.

Cause 2: out of inodes¶

Every file uses one inode. Millions of tiny files (a mail queue, session files, a broken cache, find-generated temp files) exhaust the inode table while bytes are fine.

df -i                    # IUse% at 100% is the tell
# Where are they?
sudo find / -xdev -type f 2>/dev/null | cut -d/ -f2-3 | sort | uniq -c | sort -rn | head

Fix: delete the file glut. Usual suspects: /var/spool/, PHP sessions in /var/lib/php/sessions, /tmp, an app's cache dir. You can't add inodes to an existing ext4 filesystem without reformatting; XFS allocates them dynamically.

Cause 3: files hidden under a mount point¶

Something wrote to /data before the real disk was mounted there. Now the real /data is mounted on top, hiding those files — but they still occupy the root filesystem.

# Compare mounted vs underlying
du -sh /data                          # what you see now
sudo umount /data 2>/dev/null || true
sudo mkdir -p /mnt/tmp && sudo mount --bind / /mnt/tmp
du -sh /mnt/tmp/data                  # what's actually on / underneath
sudo umount /mnt/tmp

Fix: boot single-user or bind-mount / elsewhere, clean up the shadowed directory, then remount.

Cause 4: reserved blocks¶

ext4 reserves 5% for root by default so a full disk doesn't lock out root and critical daemons. On a big data volume that's wasteful, and non-root writes fail while df shows ~5% "free".

sudo tune2fs -l /dev/sda1 | grep -i "reserved block count"
sudo tune2fs -m 1 /dev/sda1          # lower to 1% (do NOT do this on / )

Cause 5: sparse files / different block accounting¶

du reports allocated blocks; ls -l and cp may show a larger apparent size (sparse files — VM images, database files with holes). Conversely, small files still consume a full block, so du of a million 1-byte files ≫ their byte sum.

du -sh --apparent-size /var/lib/libvirt/images/*
du -sh /var/lib/libvirt/images/*        # allocated — often much smaller

A quick triage script¶

df -h /                              # is it really full?
df -i /                              # inodes?
sudo lsof -nP +L1 | grep -c deleted  # deleted-open files?
sudo du -xhd1 / 2>/dev/null | sort -h | tail    # biggest dirs on / only (-x = don't cross mounts)

Verification and troubleshooting¶

• du -xhd1 / finishes fast and the total is way below df — deleted-open files or a shadowed mount. Check lsof +L1 first (easier fix).
• df -i at 100% — inode exhaustion; du won't help, you need file counts: find <dir> -xdev | wc -l per suspect directory.
• Only /var or /tmp full, / fine — they're separate filesystems; df -h shows each. Target that one.
• Freed gigabytes with rm but df unchanged — a process still holds the deleted files. lsof | grep deleted, restart it.
• Cleared space, filled again in minutes — something is actively writing fast (debug logging, a crash loop dumping cores, a runaway dd). watch -n1 'du -sh /var/log' to catch it.

Related tools and reading¶

• On-site: SAR Report Analyzer, Linux commands reference.
• Related posts: Understanding Linux load average, LVM: grow a filesystem online.