diff --git a/README.md b/README.md index 96bcf468..8d40dbfd 100644 --- a/README.md +++ b/README.md @@ -448,7 +448,7 @@ The panel backs itself up every day at 03:15 (cron in Docker, `webvirt-backup.ti - `ssh/`: the SSH keys of the panel's user, which the SSH computes accept - `commit`, outside Docker -The database holds compute passwords and keys: the directories are readable only by the panel's user. They are on the same host as the panel; copy them to another machine as well. A failed backup is reported in `docker logs` or by `systemctl status webvirt-backup` and deletes no older backup. +The database holds compute passwords and keys: the directories are readable only by the panel's user. They are on the same host as the panel; the [off-site copy](#off-site-copy) keeps them on another server too. A failed backup is reported in `docker logs` or by `systemctl status webvirt-backup` and deletes no older backup. Take one now: @@ -459,6 +459,78 @@ systemctl start webvirt-backup && journalctl -u webvirt-backup -n 3 To restore one, follow [Restore and rollback](#restore-and-rollback) with `b` set to its directory. In Docker, copy it out of the volume first (`docker compose cp webvirtcloud:/srv/webvirtcloud/backups/webvirtcloud-backup-YYYYMMDD-HHMMSS ..`) and leave out the `git checkout` and `build` lines: the backup has no commit. To move the panel to a new machine, see [Moving to a new machine](#moving-to-a-new-machine). +### Off-site copy + +`webvirt-offsite.timer` copies the backups every day at 03:45 with [restic](https://restic.net) over SFTP to another server, encrypted with a password that only you and the panel's host hold, and keeps 7 daily, 4 weekly and 6 monthly copies there. It runs on the panel's host as root, also with Docker. It fails, and `journalctl -u webvirt-offsite` says why, when the newest backup is older than 26 hours or the copy does not finish. + +On the other server, create the user the copies go to (`wvcbackup` on `backup.example.com` below): + +```bash +useradd -m wvcbackup && install -d -o wvcbackup -m 700 ~wvcbackup/.ssh +``` + +On the panel's host, from a root shell in the directory you installed from: + +1. Install restic and curl (RHEL: `dnf -y install epel-release && dnf -y install restic`, openSUSE: `zypper -n install restic curl`) and the job: + + ```bash + apt-get -y install restic curl + install -m 755 conf/webvirt-offsite.sh /usr/local/sbin/webvirt-offsite + cp conf/systemd/webvirt-offsite.service conf/systemd/webvirt-offsite.timer /etc/systemd/system/ + ``` + +2. Give the host a key of its own for the other server: + + ```bash + install -d -m 700 /root/.ssh + ssh-keygen -q -t ed25519 -N '' -f /root/.ssh/webvirtcloud-offsite + cat >> /root/.ssh/config <<'EOF' + Host webvirtcloud-offsite + HostName backup.example.com + User wvcbackup + IdentityFile /root/.ssh/webvirtcloud-offsite + EOF + ``` + + Add `/root/.ssh/webvirtcloud-offsite.pub` to `~wvcbackup/.ssh/authorized_keys` on the other server (owned by `wvcbackup`, mode 600), then connect once to record the server's key: + + ```bash + ssh -o StrictHostKeyChecking=accept-new webvirtcloud-offsite true + ``` + +3. Write the settings with a new password and create the repository. With Docker, set `WVC_BACKUPS` to the volume, `$(docker volume inspect -f '{{.Mountpoint}}' webvirtcloud_webvirtcloud-backups)`. `WVC_HEARTBEAT` is optional: a URL called after each good copy, such as a [healthchecks.io](https://healthchecks.io) check, which tells you when a day passes without one. + + ```bash + install -m 600 /dev/null /etc/webvirtcloud-offsite.env + cat > /etc/webvirtcloud-offsite.env <&2 + exit 1 +fi +# dot files are the lock and the backup being written +restic backup --tag webvirtcloud --exclude '.*' "$WVC_BACKUPS" +restic forget --tag webvirtcloud --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune +if [ -n "${WVC_HEARTBEAT:-}" ]; then + curl -fsS -m 10 --retry 3 -o /dev/null "$WVC_HEARTBEAT" +fi