The daily backups stayed on the panel's host, so losing the host lost
them too. webvirt-offsite.timer now copies them at 03:45 with restic
over SFTP to another server, encrypted with a password only the
administrator and the host hold, and keeps 7 daily, 4 weekly and 6
monthly copies. It runs on the host as root, with Docker too, outside
the panel and its image. It fails when the newest backup is older
than 26 hours, so a stopped panel backup shows; an optional heartbeat
URL is called after each good copy.
The README has the setup, how to get a backup back from the copies,
and a fix found by the drill: moving to a new machine stopped when
the backup had no SSH keys (a panel without SSH computes).
The only backups of the panel were the ones taken by hand before an
update. Now `manage.py backup` writes a new directory under backups/
with a consistent copy of the sqlite database (the backup API, while
the panel runs, checked with PRAGMA integrity_check), the secret key,
the worker's node_id, settings.py, the SSH keys of the panel's user
and, outside Docker, the commit. It keeps the newest 7 (--keep N) and
deletes older ones only after a good backup; a failed one leaves
nothing, and a second run while one runs fails. Files are mode 600,
directories 700: the database holds compute passwords and keys. With
PostgreSQL the database is left out.
It runs daily at 03:15: in Docker from the image's cron into a new
volume, webvirtcloud-backups; otherwise from webvirt-backup.timer,
which the installer and the manual steps install. A backup has the
layout of the update backups, so the restore steps take either.
gstfsd, the root service on each compute that rewrites a shut-off VM's
/etc/shadow and authorized_keys, did what any client on its port asked.
Now it needs GSTFSD_KEY (64 hexadecimal characters, in the unit's
/etc/default/webvirt-gstfsd) and does not start without it. Each
connection gets a new challenge; the panel signs its request for it
with HMAC-SHA256 and gstfsd checks the signature before any change, so
a request recorded on one connection is refused on another. gstfsd
signs its reply for the same challenge and the panel's nonce, and the
panel accepts only that reply.
Each compute has a gstfsd key, the host's GSTFSD_KEY: the compute forms
and the API take it and never show it, and an empty value on an edit
keeps it. The bootstrap script makes the key file once, readable by
root only; the README shows how to make it and where the panel takes
it. The traffic is still not encrypted: a remote panel's port stays
open to the panel's address only.
webvirt-novncd and webvirt-socketiod get read-only system directories, no
access to /home and /root, no new privileges, no kernel tunables, modules
or cgroups, and only Unix, IPv4 and IPv6 sockets; socketiod also a private
/tmp. novncd keeps the shared /tmp, where a VM's VNC Unix socket may be.
They keep what they need: a pty and its console worker, TCP and Unix
sockets, ssh with its configuration, writes under /srv/webvirtcloud and
the service user's ~/.ssh. The units installed by the update steps bring
this.
The compute bootstrap installs gstfsd and webvirt-gstfsd.service in one
helper and no longer installs or restarts supervisor; the README's
compute step does the same, a remote panel's bind goes into a drop-in,
and a compute moving from supervisor removes its gstfsd program.
The installer and the manual installation install four systemd units
(webvirt-cloud, webvirt-novncd, webvirt-socketiod, webvirt-worker) that
run as the web server's user with absolute paths and read
/etc/default/webvirtcloud. gunicorn, socketiod and the worker stop with
KillMode=mixed, so they wind down their own children. supervisor, its
installer code and the contributed systemd files go; the README gets
the update, rollback and a one-time move from supervisor, and the
installer refuses to run over a supervisor install. Docker keeps runit;
novncd's runit script is the one-line form of the others.
The xterm console never connected: socketiod expected a different token
format, its worker had no compute credentials, one global console was
shared by every client and its output broadcast to all of them, and the
Docker image did not run socketiod at all.
- console/tokens.py: one authorize() for noVNC and socketiod; the signed
token is the only credential, checked at connect time.
- socketiod: one pty worker per client (consolecallback with the
instance id), one session per VM, a session cap, input and resize
checked; the parent never opens libvirt. The engine.io same-origin
check is on.
- consolecallback: opens the console with the compute's credentials and
exits with a message when it cannot.
- Docker runs socketiod as a runit service; nginx proxies /socket.io/
with Host and the scheme for the origin check. Supervisor and systemd
start it as well; WEBVIRTCLOUD_SERIAL_CONSOLE=1 enables it.
- Tests: token and session unit tests, socketiod as a process, and a
live test that reads SeaBIOS from a wvc-test VM on the TCP and SSH
hosts.
gstfsd only listens on loopback unless GSTFSD_BIND_HOST and
GSTFSD_ALLOW_REMOTE=1 are set; add commented environment lines for both
service definitions. gstfsd.conf now ends with a newline, which matters
because dev/libvirt-bootstrap.sh appends it to /etc/supervisor.conf.
socketiod does not validate the console token, keeps a single global PTY
and broadcasts its output to every connected client, so anyone who can
reach port 6081 can read and write a serial console that someone opened.
It is also broken for legitimate users, since console/views.py now writes
signed tokens that socketiod cannot parse.
Until it is rewritten, turn the serial console off by default:
- Add SERIAL_CONSOLE_ENABLED = False to the settings template; installs
whose settings.py lacks it are treated as disabled.
- The console view shows an error instead of the xterm page for pty
consoles while the setting is off.
- socketiod refuses to start unless SERIAL_CONSOLE_ENABLED is True.
- Supervisor: autostart=false for socketiod; drop -d (debug) from the
supervisor and systemd definitions; fix the systemd unit description;
conf/systemd/install.sh no longer suggests enabling socketiod.
- README: stop opening port 6081 and document how to enable the serial
console on a trusted network.
