rsync Patterns for Backups and Deploys

Introduction¶

rsync copies only what changed, over SSH, efficiently. Two idioms cover most ops use: snapshot backups (with hardlinks, so each snapshot is cheap) and deploys (push a build directory to servers).

The trailing slash¶

The single most common mistake:

rsync -a src/  dst/     # copy the CONTENTS of src into dst   -> dst/file
rsync -a src   dst/     # copy the DIRECTORY src into dst      -> dst/src/file

A trailing slash on the source means "the contents of". No slash means "the directory itself". Destination trailing slash doesn't matter. When in doubt, --dry-run first.

The flags that matter¶

-a        archive: -rlptgoD (recurse, symlinks, perms, times, group, owner, devices)
-v        verbose (or --info=progress2 for a single progress line)
-z        compress in transit (skip for already-compressed data or fast LANs)
-h        human-readable sizes
--delete  remove files in dst that are gone from src   (mirror)
--partial --append-verify   resume interrupted transfers
--bwlimit=10m   cap bandwidth
--exclude-from=exclude.txt
-n / --dry-run   show what would happen

-a implies -o/-g which need root to set on the destination — for a non-root deploy use -rltDv --chmod=... instead, or --no-o --no-g.

Snapshot backups with --link-dest¶

Each run creates a full-looking snapshot, but unchanged files are hardlinks to the previous snapshot — so ten daily snapshots of a 100 GB tree that barely changes use ~100 GB + deltas, not 1 TB.

#!/usr/bin/env bash
set -euo pipefail
SRC="/srv/data/"
DEST="/backups/data"
DATE="$(date +%F_%H%M%S)"
LATEST="$DEST/latest"

rsync -a --delete \
  --exclude-from=/etc/backup/exclude.txt \
  --link-dest="$LATEST" \
  "$SRC" "$DEST/$DATE.partial"

mv "$DEST/$DATE.partial" "$DEST/$DATE"
ln -sfn "$DEST/$DATE" "$LATEST"

# prune: keep 14 daily
ls -1d "$DEST"/20* | sort | head -n -14 | xargs -r rm -rf

--link-dest points at the previous snapshot; identical files become hardlinks. Deleting an old snapshot only frees space for files not linked elsewhere.

For pull-from-remote, put the rsync -a user@host:/srv/data/ ... and run the backup on the backup server (so a compromised source can't reach your backups).

Deploys¶

rsync -a --delete --checksum \
  --exclude='.git' --exclude='node_modules' \
  ./dist/  deploy@web01:/var/www/app/

# near-atomic: rsync into a new dir, then flip a symlink
rsync -a --delete ./dist/ deploy@web01:/var/www/releases/$REL/
ssh deploy@web01 "ln -sfn /var/www/releases/$REL /var/www/app && systemctl reload app"

• --checksum compares content, not mtime/size — safer for build artifacts where timestamps churn but content is identical.
• rsync updates files in place by default; --inplace avoids a temp copy (good for huge files / low disk) but a crash mid-write leaves a partial file. Default (temp + rename) is atomic per file.
• Whole-deploy atomicity needs the symlink flip.

Excludes¶

/etc/backup/exclude.txt:

- /proc/
- /sys/
- /dev/
- /run/
- /tmp/
- /var/cache/
*.tmp
*.swp
node_modules/

Leading / anchors to the transfer root. - prefix in an --exclude-from file is optional; a plain line is an exclude.

Verification and troubleshooting¶

rsync -avn --delete src/ dst/          # dry run — ALWAYS before --delete
du -sh /backups/data/*                 # snapshot sizes (hardlinks: du counts shared once)
rsync --version | head -1

• rsync copied everything again — mtime/size heuristic failed (a restore reset timestamps, a different filesystem's time granularity, or --checksum wasn't used when it should have been).
• --delete nuked the wrong things — trailing-slash mistake made the source the parent dir. Always --dry-run. Add --delete-after and --max-delete=100 as guards.
• Permission denied on destination — -a tries to set owner/group; use --no-o --no-g or --chown, or run as root on the receiving side.
• Snapshots not sharing space (each is full size) — --link-dest path is wrong/relative (must be absolute or relative to the destination), or it points at a .partial dir, or the previous snapshot is on a different filesystem (hardlinks can't cross).
• Transfer keeps restarting from zero — no --partial; add --partial --append-verify (or -P = --partial --progress).
• Slow over WAN — -z if data compresses, --bwlimit isn't the issue, check for many tiny files (rsync has per-file overhead — tar | ssh can beat it for millions of small files on the first copy).
• "some files vanished before we could transfer them" — source changed mid-run (a live directory); acceptable for backups, or snapshot the source (LVM/ZFS) first.

Related tools and reading¶

• On-site: Crontab generator.
• Related posts: Writing a Bash script that won't bite you, logrotate: configuration that doesn't lose logs.