Server backups that actually restore
Twenty years running a hosting support desk teaches you one thing above all: nobody has a backup problem until the moment they have only a backup problem. Everything below is written from the other side of those tickets.
There are two kinds of backup: the kind you've restored from, and the kind you hope works. This guide gets you from the second to the first with the minimum of moving parts.
What to back up (it's less than you think)
Not the operating system — you can reinstall that in ten minutes with the hardening checklist. Back up the things you can't download again:
- Application data — the Docker volumes / bind-mounted directories your services write to (photos, documents, uploads).
- Configuration — your compose files, Caddyfile,
.envsecrets. Small, priceless. - Databases — dumped properly (below), never just copied while running.
A useful discipline: keep everything in /srv or
/opt/stacks, one directory per service, data and compose file
together. Then "what do I back up?" has a one-word answer.
The 3-2-1 rule, translated
Three copies of your data, on two different media, one of them off-site. In self-hosting terms: the live data, a nightly backup on a second disk or machine, and a copy somewhere that isn't in your house or your VPS provider — cheap object storage (Backblaze B2, Hetzner Storage Box, Wasabi, S3) is the standard answer, at pennies per GB-month.
The off-site copy is the one people skip and the one that matters most: fire,
theft, ransomware, a provider closing your account, and a fat-fingered
rm -rf all take the local copy with them.
The tool: restic
restic is our default. Deduplicated (nightly backups of 200GB cost almost nothing after the first), encrypted client-side (your storage provider sees noise), backs up to local disks, SFTP, and every object store, and restores single files or whole trees. Borg is the equally good alternative if you prefer it; both are vastly better than tarballs and rsync scripts.
Install (apt install restic), then initialise a repository — here
to Backblaze B2, but sftp:, s3: and a plain local path
all work identically:
# credentials + repo, e.g. in /root/.restic-env (chmod 600) export B2_ACCOUNT_ID="..." export B2_ACCOUNT_KEY="..." export RESTIC_REPOSITORY="b2:my-bucket:server1" export RESTIC_PASSWORD="a-long-random-passphrase" restic init
Write the passphrase down somewhere off the server — password manager, and a paper copy. Lose it and the backups are cryptographically gone. This is the number one way people turn a good backup into no backup.
Databases: dump, don't copy
Copying a live database's files gives you a corrupt snapshot more often than not. Dump it to a file first, and back up the file. For MySQL/MariaDB in Docker:
docker exec mysql sh -c 'exec mysqldump --all-databases --single-transaction -uroot -p"$MYSQL_ROOT_PASSWORD"' \ > /srv/backups/mysql-all.sql
(--single-transaction gives a consistent InnoDB snapshot without
locking — see our transactions
article for why that works. Postgres: pg_dumpall. SQLite apps:
sqlite3 db.sqlite ".backup /srv/backups/app.sqlite".)
The nightly script
#!/bin/bash set -euo pipefail source /root/.restic-env # 1. dump databases docker exec mysql sh -c 'exec mysqldump --all-databases --single-transaction -uroot -p"$MYSQL_ROOT_PASSWORD"' \ > /srv/backups/mysql-all.sql # 2. back up data + config + dumps restic backup /srv /opt/stacks --exclude '/srv/media/cache' --tag nightly # 3. prune old snapshots (keep 7 daily, 4 weekly, 6 monthly) restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune # 4. verify repository integrity (a small random sample — cheap enough nightly) restic check --read-data-subset=5%
Schedule it: sudo crontab -e →
30 3 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1.
Then — the step everyone skips — make failure loud: pipe the
result to a healthchecks.io ping, an ntfy notification, or an email. A backup job
that has silently failed for four months is the standard shape of the disaster
ticket.
The restore drill
Untested backups are a hypothesis. Once a quarter, on a scratch VPS or a spare machine:
restic snapshots # see what you have restic restore latest --target /tmp/restore # or a specific snapshot ID # then actually start a service from the restored data and log in
You are testing three things: the backups contain what you think, you know the commands under mild pressure, and the passphrase you wrote down is right. Time the whole exercise — that number is your real recovery time, and it should fit in an evening. If it doesn't, simplify until it does.
Two traps to end on
- RAID, snapshots and sync are not backups. RAID survives a disk failure and faithfully mirrors your deletions; ZFS/Btrfs snapshots are wonderful but live on the same machine; Syncthing/Nextcloud replicate your mistakes to every device within seconds. Useful layers — not the off-site copy.
- Backing up the VPS through the provider's snapshot button alone. Convenient, worth having — and gone the day the account is suspended, the provider folds, or you miss a bill. One copy must live somewhere that provider doesn't control.