echodebug compared _ECHO_DEBUG and BS_TRUE with -eq, but neither is set
anywhere, so each step printed "[: : integer expression expected". It
now prints debug lines only when _ECHO_DEBUG=1 is set.
Files that libguestfs creates have no SELinux label. On an enforcing
guest (RHEL, Rocky, Alma, Fedora) sshd may not read the unlabeled
/root/.ssh/authorized_keys, so a key added from the panel did not let
anyone log in ("avc: denied { read } ... unlabeled_t").
After writing a key, gstfsd now relabels /root/.ssh with the guest's own
file contexts (/etc/selinux/<SELINUXTYPE>/contexts/files/file_contexts),
as virt-customize does. Guests without an SELinux policy or with SELinux
disabled are left as they are. If the guest needs labels but the host's
libguestfs cannot set them, the request fails before anything is written.
A root password needs no labelling: /etc/shadow keeps its file and label.
The host commands hard-coded www-data, which does not exist on RHEL and
openSUSE (the panel runs as nginx there), so the first command failed
with "invalid user". The user is now set once in U and its home is read
with getent, since ~$U is not expanded. The host block also ends with
the same connection check as the Docker one.
Both blocks are labelled, and the step says to use only the one for the
installation type: a key set up on the host is not seen by the panel in
Docker.
AGENTS.md is a local agent file and is ignored.
The README gave host commands (sudo, ~www-data) and one sentence for
Docker, where there is no sudo and exec runs as root: a key made that way
lands in /root/.ssh, which the panel (www-data) does not use, and the
first connection fails with "Host key verification failed". Docker now
has its own commands, run as www-data, ending with a connection check.
Both variants accept a new host key once (accept-new) instead of
disabling host key checking.
Two tests released a hanging open and slept 0.1 s before expecting its
connection; on a slow CI runner the open had not published it yet and
the stored timeout error came back. They now wait until the connection
is published.
A clone copies the disks of the VM's persistent definition, but its disk
quota and the disk names chosen with automatic naming came from the
running VM's disks, which differ after a disk is attached or detached on
the running VM only.
The Destroy button was disabled and the confirmation page refused a
paused VM ("Instance is suspended, cannot destroy!"), while the destroy
request itself stops a paused VM like a running one. Both now offer it.
The form required a bridge member for an ethernet interface, which does
not use one. When the host's network devices could not be read, the error
was swallowed and every creation failed with "the device is not on this
host"; the page now shows the real error and does not try.
Without instances.snapshot_instances (snapshots) or is_vnc (console
settings) the request was accepted and silently did nothing, although
the documented answer for a missing right is 403. delete_external_snapshot
also asked libvirt for the VM state before checking the permission.
Redirects to #access (root password, SSH key) and other top-level panes
opened nothing, and some sub-tab ids did not exist. static/js/instance_tabs.js
opens the pane and the tab that holds it from the page structure, and
opens Settings on its first section, without leaving the current tab;
guest OS info is loaded whenever it is the section opened. The bottom bar linked to
host pages, which only administrators may open, and showed host names to
everyone. A stray </li> is removed; the page's inline scripts and the tab
logic are checked with Node.
Power cycle force-stopped and started a template, which power on refuses.
A non-staff owner with is_delete could delete a template that the
template rule otherwise leaves to superusers and staff owners;
get_instance(perm_type="delete") now refuses it, and the page offers
Destroy by the same rule.
A VM operation (or destroy and clone, which hold the whole compute) waits
for its compute's lock while a migration holds it. When the migration
moved the VM's record to the destination, the waiting operation went on
there holding only the source compute's lock, unordered with a
reconciliation, migration or disk attach on the destination. After taking
the locks it now checks the VM is still on the locked compute and
otherwise answers busy, so the retry locks the right one. Clone also
checks again under the lock whether the user may clone the VM, and
checks the quota and the clone's settings against the VM as it is then:
its template mark, size or the user's rights may have changed meanwhile.
gstfsd mounted every partition it listed: a root on LVM was never found,
a disk with two OS roots got the change twice and two concatenated
replies, and no root, an error or a request longer than 1024 bytes ended
the connection without a reply. It now lets libguestfs inspect the guest,
mounts its one OS the way the guest does (a broken filesystem off the
changed path is left out), and sends exactly one reply, keeping the first
error. The root entry is found on any line of /etc/shadow and must be
unique; a key needs no shadow file. A request is at most 64 KiB. Each
change runs in a worker process with a directly started appliance; both
are killed when the request is not done 100 s after gstfsd accepted it
(GSTFSD_DEADLINE may only lower that), before the panel's 120 s timeout
releases the VM, or when the panel's connection ends, and the reply waits
until they are gone, so nothing touches the disk once the panel has given
up. A worker that dies says why. The panel
connects over IPv4 or IPv6, sends the whole request and reads the reply
until gstfsd closes.
The documented rule held only for snapshots: an owner with is_change could
still resize a template, set its root password and SSH keys, change its
options and console settings. get_instance(perm_type="change") now
refuses a template to owners who are not staff, which covers every change
view and the API; the per-view snapshot checks it replaces are removed,
and the page offers those controls by the same rule. Viewing, the console,
power (a no-op on a template) and cloning are unchanged.
The page showed controls the backend then refused: power to global
viewers, the console, virt-viewer and VDI controls and the root password
and SSH key forms to viewers, resize to staff, per-vCPU hotplug, video
model and guest agent to non-superusers, clone and options by the clone
permission alone, snapshots without change permission. The page now asks
the views' own rules (owner, is_change, clone and snapshot permissions,
the template rules). OS info, opened from the header, is there for every
viewer; the XML editor and tab restoring after a redirect no longer throw
when a pane is absent; Settings opens on the first section the user has.
The authorization matrix also covers a template VM, the template clone
rule and the template snapshot rule.
The IPv4 address and gateway were required even for DHCP or no IPv4, so
such an interface could not be created; malformed addresses passed and a
missing prefix crashed the XML builder. They are validated like the IPv6
fields now (netmask notation accepted, IPv6 scope ids rejected), and a
static address without a gateway gets no empty route. Interface names
follow the Linux rule (up to 15 letters, digits, . _ -). A bridge member,
or the device an ethernet interface configures, must be a device of the
host (the page no longer falls back to made-up names), a bridge cannot
contain itself, and needs STP and a non-negative delay.
The README grew as a list of per-change notes. Reorder it along the
operator's path (install, first login, compute node, configuration,
update, usage, development) and correct what no longer matched the code:
supported distros and Python, the openSUSE nginx step that installed a
full nginx.conf as a vhost, the disabled serial console listed as a
feature, missing ALLOWED_HOSTS and file-mode steps, services not
restarted, and an update path that copied settings blocks by hand
instead of rebuilding settings.py from the template with the panel
stopped.
The first appsettings migrations insert the default rows with fixed ids,
which a PostgreSQL id sequence does not see; the login lockout migration,
the first to insert without an id, then collided with id 1. It now moves
the sequence past the existing rows first (a no-op on sqlite), which also
covers a database upgraded from any earlier migration. A test compute
password exceeded its 14-character column, which only PostgreSQL enforces.
Reviewing the error handlers outside instances/ turned up three that masked or
swallowed the failure they caught:
- appsettings/settings.get_settings() runs on every request (middleware) and
loads app_settings from the database. Its bare `except: pass` left `entries`
unbound on a database error, so the next line raised UnboundLocalError
instead; it also caught KeyboardInterrupt/SystemExit. It now catches
DatabaseError and returns, keeping the already-loaded values.
- vrtManager.network.refresh_dhcp_leases raised a formatted string
(`raise f"..."`), which Python rejects with TypeError, hiding the real
DHCPLeases failure. It now raises util.OperationError, which the API/HTML
error handling already renders as a libvirt error, and clears self.leases.
- storages.views.get_volumes caught libvirtError and answered 200 with an empty
body, so the caller could not tell a failed pool read from an empty pool. It
now returns 400 with the error.
The other broad handlers reviewed (appsettings/nwfilters/virtsecrets views,
storages/upload cleanup and recovery paths, vrtManager display getters) report
the error or are best-effort and were left as they are.
Tests: get_settings() survives a database error during the query and during
iteration without raising and still loads a real value; refresh_dhcp_leases
raises OperationError (keeping the cause) not TypeError; get_volumes returns 400
on a listing failure and the sorted volumes on success. The tests fail on the
previous code.
Several instance mutation views crashed with a bare HTTP 500 when a form field
the UI always sends was absent or malformed (a direct or crafted request);
ExceptionMiddleware turns only libvirt and timeout errors into 4xx, so these
reached a 500. Fixed:
- add_network parsed a None network source ("'NoneType' has no attribute
'split'"); it now requires a selected network.
- set_qos read an unbound local 'mac' when no field name contained "mac", and
passed an invalid direction or non-numeric rate to the proxy (ValueError); it
now requires a MAC, an inbound/outbound direction and whole-number rates.
- unset_qos passed an empty direction into the proxy's XPath (XPathEvalError);
it now requires a MAC and a valid direction.
- resizevm_cpu and resize_memory ran int() over an empty or non-numeric value;
they now validate with int() (not str.isdigit(), which accepts "²" that
int() rejects).
- add_public_key passed a missing/non-numeric sshkeyid into an integer-id
lookup (ValueError); it now requires a numeric id.
Each reports the problem and returns to the VM page without calling libvirt or
logging, as change_network already does.
Tests: each view with a field missing, non-numeric or a unicode digit returns
302 with an error message, writes no log, and does not attempt the mutation.
The tests fail on the previous code.
OTP (OTP_ENABLED) was enforced only by the HTML login form, but two other
doors let an enrolled user authenticate with a password and no token, so a
leaked password defeated the second factor. Both were reproduced with a
confirmed TOTP device: /api-auth/login/ (DRF's browsable-API login) created a
site-wide session, and DRF's default BasicAuthentication let every /api/v1/
endpoint accept HTTP Basic credentials.
The api-auth/ include is removed, so the only interactive login is the
OTP-aware form at /accounts/login/; the browsable API still works with the
session that form sets. REST_FRAMEWORK now pins DEFAULT_AUTHENTICATION_CLASSES
to SessionAuthentication only, so the API requires a session and no longer
accepts a password over HTTP Basic. A session therefore implies the OTP factor
was met. Password-based programmatic API access is intentionally dropped.
The authorization matrix drops its rest_framework:login/logout allowlist
entries (the routes are gone). New tests assert the browsable login route is
removed, HTTP Basic is rejected on the API, Basic is not a default auth class,
and a session still reaches the API. Verified live with OTP_ENABLED that
/api-auth/login/ is 404 and an API Basic request is 403.
With OTP_ENABLED the login used django-otp's OTPAuthenticationForm. Under the
pinned django-otp 1.7.3 that form could not log anyone in through the shipped
page: the page posts only a 6-digit otp_token, but the form now needs the
device named explicitly, so an enrolled user got "Please select a device.",
and a user who never enrolled got "Please enter your OTP token." with no
device to produce one. Verified in an isolated test database.
SingleDeviceOTPForm makes OTP per user: a user with a confirmed TOTP device
must enter a valid code; a user without one logs in with the password alone.
The form chooses the device itself (the user's oldest confirmed TOTP device,
locked with select_for_update inside clean_otp's transaction so replay and
throttling counters cannot race) and ignores any posted otp_device, which
django-otp does not check for confirmed status or type. The login page drops
the HTML "required" on the token field (the field is optional for an
unenrolled user) and says so.
The "Lost OTP?" page (email_otp) created a confirmed device when none existed.
With per-user OTP that let anyone who knows an address turn a password-only
account into an OTP-required one and, if the mail never arrived, lock it out.
It now resends the QR only to a user who already enrolled and never creates a
device (get_existing_totp_device); admin enrollment (admin_email_otp) still
creates one. Email QR distribution and confirmed-on-create enrollment are
unchanged by user decision.
Tests: per-user login (no device -> password only; device -> valid token
passes, wrong/empty token fails), a posted or another user's device is not
used, a token cannot be replayed, and the real login page over HTTP with
OTP_ENABLED; email_otp sends nothing and creates no device for an unenrolled
user. The new tests fail on the previous code.
A compute's hostname and login go unquoted into the libvirt connection URI
(qemu+ssh://login@host/system, qemu+tcp://host/system, qemu+tls://...) and,
for an SSH compute, into the ssh command line of the DRBD status view. They
were barely validated: validate_hostname used re.match with a negated class,
so it caught only a leading special character, and login was never checked.
A hostname like "192.0.2.1/system?command=/path" injected a libvirt URI
parameter (verified: connecting ran the named command through the ssh
wrapper), and TLS/SSH parameters such as no_verify, pkipath, socket or
netcat could be added the same way; a login like "-oProxyCommand=..." became
an ssh option. The DRF ComputeSerializer applied neither check, and
validate_name tested a string literal instead of its argument, so it never
raised.
validate_hostname now accepts only an IPv4 address or a host name with
fullmatch (no scheme, port, path, query, whitespace or IPv6 literal; IPv6 is
left out until the URI building brackets it). validate_login accepts an empty
value (TCP falls back to the server's authentication, the socket transport
ignores the login) or a plain name with no leading "-". Both sit on the
Compute model fields, so the forms and the DRF serializer (which copies model
validators onto its fields) reject the values; a migration records the
metadata. validate_name checks its argument. drbd_status validates the login
and hostname before building login@host and ends ssh option parsing with
"--"; an existing or bypassed bad row is skipped rather than run.
Tests: hostname/login validators accept real values and reject URI and
option injection; the SSH form and the serializer (create, and an empty TCP
login) enforce them; drbd_status skips an injecting target. Checked the real
test computes still validate. The new tests fail on the previous code.
libvirt keeps a VM's UEFI variables (NVRAM) and emulated TPM state in
files of its host. A live migration carries both: on the test hosts
(libvirt 11.10, QEMU 10.1) the destination got the same NVRAM content
once the VM ran and the same TPM state, and the source TPM state was
removed. Moving a shut-off VM moves only its definition, so neither
went along: on its first start on the destination libvirt made a new
NVRAM from the template and a new TPM. Boot entries and enrolled keys
are lost, and disks sealed to the TPM no longer unlock.
A shut-off VM whose definition has an NVRAM file or an emulated TPM is
now refused before anything changes: "Start the VM and migrate it
live." A passthrough or external TPM keeps no state on the host and
still moves. The check sits in moveto, not in the define/undefine
workaround, as libvirt's own offline migration does not carry the files
either. The keep-NVRAM and keep-TPM undefine flags of that workaround
are gone, as no VM that needs them reaches it.
Checked on the test hosts: a shut-off UEFI VM with a TPM is refused and
stays on the source with nothing defined on the destination; the same
VM migrates live. The new tests fail on the previous code.
A VM that changes its memory faster than the network carries it does
not finish a live migration without auto converge (measured: a guest
rewriting about 1.2 GB/s did not finish in 90 s over an 8.5 Gbit/s
link, and finished in 16-18 s with it). Auto converge only slows the
VM's CPUs while the migration cannot keep up, so a calm VM is not
affected.
The migrate form ticks it for a running VM. The API already documented
it as on by default but read a left-out option as off; the serializer
now has the defaults (auto converge on, compress and unsafe off) for
JSON and form-encoded requests, and the schema shows them.
A VM that changes its memory faster than the network carries it never
finishes a live migration without auto converge: measured with a guest
rewriting about 1.2 GB/s over an 8.5 Gbit/s link. The migration then
ran until gunicorn ended the request after 600 s. libvirt cancelled it
when the request went away and the VM stayed on the source, but the
user got an error page and nothing was logged.
The app now cancels a migration that has not finished after 540 s
(MIGRATION_TIME_LIMIT, below the 600 s of gunicorn.conf.py; a test
keeps it below). The failure is reported and logged like any other:
the migration was cancelled, the VM stays on the source host, and a
VM like that needs auto converge. A timer that fires as the migration
ends finishes before the compute locks are released, and an error
the cancel did not cause keeps its own message.
With "VM Clone Auto Migrate" on, a new clone was migrated to a random
compute, whatever storage, CPU or network that host had, and a failed
migration showed as a failed clone although the clone existed.
The setting and its code are removed; migration appsettings.0015
deletes the setting. A clone can be migrated afterwards like any VM.
The README's update notes list the migration changes: the migration
address, storage pools for the disk check, and this setting.
What migrate_instance did around the migration itself:
- It checked the destination with a 1 s TCP probe of the libvirt port,
which passes while libvirt is down or refuses the login. It now uses
the compute's connection status, as the rest of the app does.
- Two requests for the same VM waited for the compute locks one after
the other; the second then migrated with the VM's old location. The
VM's record is read again under the locks, and the request is
refused when the VM is no longer on the source.
- Restoring autostart on the destination came after the migration but
before the database move; when it failed, the VM ran on the
destination while the app recorded it on the source. Autostart is
now read from libvirt and a failure to restore it is only logged.
- A database failure after a successful migration was logged as
nothing; the log now says the VM migrated and recording it failed.
- When the destination's list had already made a record for the VM,
the merge kept the owners but dropped the template mark.
Migration needs the VM's disks on storage both hosts reach, and nothing
checked that. A VM with a disk the destination did not have failed
during a live migration; a shut-off VM's definition moved and could not
start there.
moveto now refuses the migration before anything changes, naming each
disk (target and source) that is not a volume of the destination's
active storage pools. Checked: the disks, CD-ROM images and block
devices of the live and the persistent definition, pool volumes, and
backing files, as the definition or the destination's volume lists
them (a shut-off VM's definition does not). Pools are refreshed before
a disk counts as missing, so a file another host made on shared
storage is found. Network disks and network backing images are
reached the same way from every host and are not checked; disk types
that cannot be checked (dir, nvme, ...) are refused.
This checks that the destination knows a volume, not that it is the
same file: shared storage should be a storage pool on each host. A
shared file outside any pool, or a block device no pool covers, is
refused.
moveto read the VM state for each flag and dropped options that did
not fit it, so a request could run as something else than asked; the
log then named the mode from the form ("live" or "offline"), not the
one used. The API did not log at all and answered "instance migrate is
started" after the migration had finished.
- moveto reads the state once. A running VM migrates live, or without
live paused for the transfer (non-live); a paused VM stays paused; a
shut-off VM moves offline. Live for a shut-off VM, offline for a
running or paused one, and any other state are refused before
anything changes. unsafe, auto converge (live only) and compressed
apply to a running VM.
- migrate_instance logs the outcome for the VM page and the API: the
mode and options used, or the error. The API answers
"instance is migrated (<mode>)" and accepts requests that leave out
an optional option.
- Post copy is gone from the form, the view and the API: only the
flag was set, and without migrateStartPostCopy, which the app never
called, the migration never switched to post copy.
libvirt's offline migration fails on some builds with "operation
failed: domain is no longer running" (RHEL-156800: Rocky/RHEL 9.8
libvirt 11.10.0-12 has the change that broke it, not the fix, which is
upstream in 12.2.0). A shut-off VM could not be migrated at all there.
A shut-off VM is only its definition, as the disks are on storage both
hosts share, so moveto now moves that itself:
- refused first when the VM has snapshots or a saved state, or the
destination already has a VM with its UUID or name;
- defined on the destination from the source's migratable XML, then
undefined on the source; NVRAM and TPM state stay on the source;
- after a failure the VM stays defined on one host: the destination
definition is removed only while the source still has its own, and
a VM another client started on the source meanwhile gets its
definition and autostart back there.
The destination's qemu migration hook does not run for it. Running and
paused VMs still migrate through libvirt.
Migration is client-driven: libvirt sends the VM's memory to the
destination's own host name unless a migration URI is given, and the
app never gave one. Where the hosts cannot resolve each other's names
(or the names point at a management network) every migration failed
with "Unable to resolve address '<host name>' service '49152'".
A compute now has an optional migration address (an IP address or a
host name, no scheme or port), editable in the compute form and the
API. A VM migrating to that compute is sent to tcp://<address>
(bracketed for IPv6). Empty, the default for existing computes, keeps
the previous behavior. The compute's host name is not reused, because
for SSH computes it can be an ssh_config alias.
The migration log entry names the address when one is set.
Migration left the VM defined on the source host unless "Delete
original" was ticked (VIR_MIGRATE_UNDEFINE_SOURCE). With the shared
storage migration needs, that definition points at the disks the
destination VM is running on: the VM showed on both hosts and could be
started (or autostarted) a second time against the same disks; only a
configured lock manager stops that.
moveto now always passes VIR_MIGRATE_UNDEFINE_SOURCE with
VIR_MIGRATE_PERSIST_DEST. The option is gone from the form, the view,
the API (MigrateSerializer.xml_del) and the clone auto-migration. It
removes the definition, not the disk files.
The TCP and TLS compute forms never show the stored password, so the
field is empty on every edit. Saving the form without typing the
password again replaced it with nothing, and the next connection to the
host failed to authenticate.
An empty password field on an edit now keeps the stored password; a
new value replaces it.
