The first page of the create wizard built the next page's URL from a
reversed ".../x86_64/pc/" by replacing "x86_64" with the chosen
architecture and then the first "pc" with the machine type. An
architecture containing "pc" got the machine type inside its name:
ppc64le with pseries went to /instances/create/1/ppseries64le/pc/, so
VMs could not be created on PowerPC hosts.
The URL is now reversed with the placeholders __arch__ and __machine__,
which the script replaces with the URL-encoded values.
The test renders the page, runs the script with node and fails on the
previous template for ppc64le.
When a VM action failed because its host could not be reached (the
connection could not be opened, or it dropped during the call), the
error page showed libvirt's connection error to any user: it names the
host and how it is reached, e.g. "ssh: connect to host 10.x port 22".
The lists and the VM page already keep that to administrators.
- Opening a connection now fails with ConnectionFailed (a subclass of
OperationError). A dropped connection is recognized by its error
domain and code (RPC/remote/SSH layer with a transport code, measured:
a keepalive timeout is VIR_FROM_RPC/VIR_ERR_INTERNAL_ERROR); an
operation the daemon does not support (VIR_ERR_NO_SUPPORT) is not
treated as one.
- ExceptionMiddleware (pages and API) and the views that report the
error themselves (snapshot, revert, disk deletion, clone) show it to
superusers and view_instances users, and to everyone else "The host
of this virtual machine cannot be reached right now." Errors of the
operation itself ("domain is not running") are shown as before.
Checked on the test setup: powering on a VM of an unreachable compute
shows the owner the generic text and the administrator the error. The
tests fail on the previous code.
Four lookups built XPath expressions from values that come from
requests (CodeQL py/xpath-injection, plus get_capabilities, which
CodeQL did not flag; the pages are for superusers):
- The architecture in the create wizard's URLs went into
arch[@name='...'] in get_emulator, get_machine_types and
get_capabilities: "nope' or '1'='1" matched every architecture. It is
now an XPath variable ($arch), which is never parsed as XPath.
- Network filter rules were found with a predicate built from the
posted action, direction and priority: a quote raised SyntaxError (an
error page). The rule is now found by comparing its attributes.
- unset_qos put the posted direction into the path, so "*" removed
every bandwidth setting. It accepts inbound and outbound, as set_qos
does; the network page shows an error for anything else.
No other XPath in vrtManager is built from a value; the remaining
expressions are constants. The tests fail on the previous code.
The settings page lists the themes of SASS_DIR and writes wvc-main.scss
into it when the theme changes; the value could be any path, so a
superuser could make the app list and write files anywhere it may
(CodeQL py/path-injection).
SASS_DIR is now accepted only if it resolves to an existing directory
inside the project (the default is dev/scss). A stored value outside it
is not used: the page shows an error and offers no themes, so nothing
is written. Installations that keep their SCSS outside the project
need to move it under it.
The tests fail on the previous code for an absolute path, "../",
a path climbing out and a missing directory, and for a stored value
outside the project.
The settings page took the posted theme name as it came and wrote it
into the @import lines of wvc-main.scss and the compiled CSS: a name
with "../" or a quote imported files outside the themes directory
(CodeQL py/path-injection; the page is for superusers).
The name must now be one of the directories in SASS_DIR/wvc-themes, the
list the page already offers; anything else is refused with a message
and nothing is written. themes_list is empty when SASS_DIR has no
themes, so every name is refused then.
Checked with the real dev/scss: flatly is applied and compiled as
before, "../../etc" is refused and leaves the CSS alone. The test fails
on the previous code.
The test and lint workflows set no permissions, so their GITHUB_TOKEN
had the repository's default access (CodeQL
actions/missing-workflow-permissions). Both only check out and build
the code: they now ask for contents: read.
.github/workflows/codeql-analysis.yml was disabled on GitHub; code
scanning runs through GitHub's default CodeQL setup instead. The file
only kept the old analysis category (and its 2023 alerts) around.
The expression that checks a storage pool's target directory nested a
repetition inside another, so a crafted target backtracked
exponentially: "," followed by 22 dashes and "!" took 1.3 s, and every
two more characters four times as long. The field allows 100
characters, so one form post could hold a worker for good (CodeQL
py/redos; the page is for superusers).
The new expression has no nested repetition and accepts exactly the
same targets: compared with the old one on 200,000 random strings over
letters, digits, "/", ".", "-", "_", ",", "!", space and newline, they
agree on every string. A crafted 100-character target is now checked
in microseconds. The tests pin the accepted and refused targets and
time the crafted one.
The user edit page redirected to its ?next= parameter after saving,
whatever it held: a link to the edit page with next=https://other.site
sent the administrator who saved the form to that site (CodeQL
py/url-redirection). The account page links there with a next of this
site.
next is now followed only when url_has_allowed_host_and_scheme accepts
it for this host, as user_instance_delete already does; otherwise the
user list. The test fails on the previous code for an absolute URL, a
scheme-relative one and a backslash variant.
Templates do not count towards a user's quota and cannot be started.
Staff owners may mark and unmark their VMs as templates, so a staff
user could mark a VM as a template, use the freed quota elsewhere, and
unmark it again: the VM came back without any check, over the quota.
For users with a quota (everyone but superusers), turning a template
back into a VM now passes check_user_quota with the VM's size (one VM,
its vCPUs, memory and disks). If the quota is reached or cannot be
verified, it stays a template and the user is told why. Marking a VM as
a template frees quota and is not checked. The view takes the
per-user quota lock like the other quota-checked changes.
The existing template-flag test gives its mocked VM a size. The new
tests fail on the previous code.
The VM page's getters read the domain XML on their own: one load made
86 libvirt calls for a real VM, 29 of them for the same three XML
documents (live, live with secrets, persistent) and 6 for the host
capabilities. Each call is a round trip to the host.
wvmInstance.cached_reads() is a block in which _XMLDesc (per flags
value) and get_cap_xml are read from libvirt once; it ends with the
block, even on an error, does not nest, and _defineXML drops it.
Outside a block every read goes to libvirt as before. Only the VM page
view, which does not change the VM, uses it, around its rendering.
Views that change the VM keep reading fresh XML: most changes (devices,
QoS, memory) do not go through _defineXML, and several views read,
change and define more than once in one request.
Measured on the test host: 86 -> 55 calls (XMLDesc 29 -> 3,
getCapabilities 6 -> 1) for one VM, 82 -> 54 for another; the page took
0.23-0.25 s before, 0.14-0.15 s after. The HTML of both pages is
identical before and after (CSRF tokens aside). A test renders the page
on libvirt's test driver through the real getters: 25 XML reads before,
one per kind now.
django-bootstrap-icons serves icons from static/icon_cache and fetches a
missing one from the CDN while the page renders; a server without
internet access shows an error text instead of the icon.
- copy (Clone on the instance lists), envelope-open ("Lost OTP?" page)
and save (VM settings, network page) were not cached; they are now,
fetched with django-bootstrap-icons itself.
- The user and group lists asked for 'exclamation-triangle ' with a
trailing space, an icon that does not exist; the warning icon never
showed.
A test checks that every {% bs_icon %} in the templates has its cache
file; it fails on the previous files.
/accounts/email_otp/ mails a user their OTP QR code and needs no login.
Measured with the default settings (OTP disabled):
- It was reachable although OTP was off, and created OTP devices.
- Every request sent a mail: 20 requests for one address, 20 mails, so
anyone who knew an address could fill that mailbox.
- An address shared by two users, or a mail server that could not be
reached, ended in an error page (500).
- Inactive users were mailed too.
The page now exists only while OTP_ENABLED is set. It mails a single
active user whose address matches (case-insensitively), at most once
every 5 minutes per user (Django's cache: with several workers the
bound is per worker), and logs a failed delivery. The answer is the
same in every case, so it does not reveal whether an address belongs
to a user.
The QR codes come from django-qr-code (segno); both the mail and the
account page render them without the qrcode package removed earlier.
The new tests fail on the previous code.
The instance lists, the VM page header and the power tab knew three
libvirt states: running, paused and shut off. A VM that was blocked,
shutting down, crashed, suspended by its guest or without a state got
an empty status cell and no actions, so a crashed VM could not even be
forced off from the list. The lists also offered actions users could
not use: Suspend and Resume (views for superusers) to everyone, and
power actions to users who see every VM through the read-only
view_instances permission.
instances/states.py holds the rules in one place: the label and color
of each state, the actions each state allows, and which of them a user
may use (superusers all; a VM's owners all but suspend and resume, and
nothing on a VM an administrator suspended, as the VM page already
showed; nobody else any). Blocked is a running VM; shutting down,
crashed and suspended by the guest get Force Off, after which the VM
can be started. A shut-off template offers Clone to whoever has the
clone permission, as the clone view allows.
The templates read them through two tags (vm_states): the lists and the
VM page header show the state, instance_actions.html has one slot per
button, each enabled or disabled from the allowed actions. The power
tab offers Force Off in the states that need it.
The new tests check the rules directly and render every state in the
lists and on the VM page for a superuser, an owner and a
view_instances user; they fail on the previous code.
The admin page that creates a user saved the form first, which wrote
the user with the raw password in the password column, and hashed it
with a second save afterwards. The raw password was in the database,
and in its journal and any copy taken meanwhile, and stayed there if
anything failed between the two writes.
- UserCreateForm hashes the password before the first save; groups
and permissions are saved as before.
- The form applies AUTH_PASSWORD_VALIDATORS, after the user fields are
set (so the similarity check sees the new username), as the password
change forms already did. The settings template configures no
validators, so nothing changes unless an installation sets them.
The tests record the SQL parameters of the request: the previous code
wrote the raw password in the INSERT; it also accepted passwords that
configured validators reject.
The VM page polls the VM state every 5 s and, while its Stats tab is
open, the statistics every 10 s; the compute overview polls the host
graph every 5 s. All used setInterval:
- A new request went out on schedule while the previous one was still
open. When a host goes silent, a stats request waits up to 30-45 s,
so each open page held several sync workers at once.
- Each time the Stats tab was shown, the page made new charts and a
new interval without stopping the old ones, and polling went on on
the other tabs of the page.
- Hidden pages kept polling at full rate.
Measured in a browser against a test server (old code / new): Stats tab
shown 4 times, 31 s on it: 12 / 2 requests; 31 s on another tab of the
page: 12 / 0; page hidden for 31 s: 15 / 0, and the new code asks once
right after the page is shown again; with 25 s answers, at most 15 / 1
requests open at once.
static/js/poll.js repeats a request while the page is visible: the
next one 5 or 10 s after the previous answer, none while hidden, one at
once when shown again. The Stats tab makes its charts once and polls
only while shown. The answer handler runs after the poller's own
bookkeeping, so an error in it (a chart for a disk added meanwhile)
does not stop the polling. A Node test of poll.js runs with the test
suite when node is installed.
Each open VM page asks for its statistics every 10 seconds. For a
running VM the stats view measured the CPU, then each disk, then each
NIC, sleeping one second for each. With two disks and a NIC a request
held a sync gunicorn worker for over 4 s, so a few dozen open VM pages
could keep every worker busy.
wvmInstance.usage() replaces cpu_usage, disk_usage and net_usage, the
stats view's only callers: it reads all counters, sleeps one second
once and reads them again. Values and units are unchanged (CPU percent
of the host, disk bytes, network bits per second); compared against the
previous methods on the same counters, the results are identical.
Measured on the test host for a running VM with two disks and a NIC:
the stats request took 4.1-4.3 s before, 1.1-1.3 s now. The new test
fails on the previous code.
A VM owner who may manage the console (is_change and is_vnc) could set
its listen address to all interfaces and remove its password: a VNC
console reachable from the network without a password. Checked on a
test host: the owner's form changed the VM's listen address to 0.0.0.0
and then removed the password.
- The listen address field is disabled for users who are not
superusers; a disabled field ignores posted values. They can still
set or change the password and the keymap.
- A console whose persistent definition does not bind it to loopback
(or a Unix socket) only loses its password through an administrator.
A binding the XML does not pin down counts as not local: listen type
network, or no address (the host's default in qemu.conf).
On the test host, the owner's form now leaves the listen address alone
and is refused when it removes the password of a console listening on
0.0.0.0; with the console on localhost the password can be removed.
The new tests fail on the previous code.
The connection manager registered libvirt's event implementation and
started the event loop thread when vrtManager.connection was imported.
novncd imports it and then forks a child per console (websockify's
multiprocessing), so every child was forked from a process with a
running libvirt thread, which libvirt does not support, and the child
had no loop thread of its own: keepalive never ran there. Measured in
the test setup: in such a child, a call on a host that went silent
still waited after 70 s.
- The loop starts with the first get_connection of a process. The
state is per process and shared by all managers; libvirt's event
implementation is registered once and a child inherits it.
- After a fork, the child renews the lock and starts its own loop on
first use. It stops using the inherited connections, which are the
parent's sockets, but keeps them: releasing them would run libvirt
destructors in the child. No process connects before forking today
(novncd's parent does not connect, gunicorn does not preload); this
keeps such a case safe.
With the change, novncd's parent has no libvirt thread, the console
child starts its own, and the same call fails after 29 s with a
keepalive timeout. A test forks a real child; it fails without the
fork handling.
The VM page reads the VM from libvirt throughout. When its host could
not be reached, the connection error replaced the page with the error
page and showed libvirt's message, which names the host and how it is
reached, to any user who may view the VM.
The page now checks Compute.status first. If the host is not connected,
it goes to the instance list with a message: administrators (superuser
or view_instances) get the host and the error, other users only that
the VM's host cannot be reached right now. The list shows the VM as
unknown.
This covers the page itself; actions on the VM still report libvirt's
error if the host drops. The detail page tests now mock a connected
host.
Checked against the test hosts: a real VM's page opens as before, a VM
on an unreachable compute leads to the list with the user's or the
administrator's message. The new tests fail on the previous code.
check_user_quota added up the vCPUs, memory and disks of the user's
VMs, but skipped a VM whose host failed a TCP probe. A host that was
down therefore let users exceed their quota, and the probe always failed
for ssh computes named by an ssh_config alias or reached through
ProxyJump, so their VMs were never counted. A VM that could not be read
turned the page into the error page instead.
- Hosts are checked with Compute.status, the connection WebVirtCloud
uses, once per host. A VM that cannot be read (host down, VM gone)
makes the usage unknown: clone and resize up are refused with their
own message (fail closed).
- A request that adds nothing is not checked, so a user over quota can
still shrink a VM. Users without limits, and an exceeded instance
count (from the database), need no host either.
- Memory resize of a running VM changes only the current memory, but
the quota was charged with the submitted maximum. It is now charged
with no change, and resize_mem gets the current maximum, so a VM that
shuts down meanwhile cannot take an unchecked one.
- The VM page no longer computes the quota on every load (it read every
VM of the user on every host for a "quota reached" banner). The limit
is reported when a change is refused.
The test that expected a down host to pass the quota now expects it to
be refused. Checked against the test hosts: with a VM behind an
ssh_config alias the previous code counted 29 vCPUs, the new one 31;
with a VM on an unreachable host the previous code allowed +1 vCPU,
the new one refuses it after 5.1 s and allows -1 at once. The new
tests fail on the previous code, except the ones guarding unchanged
behaviour.
The instance list showed every user an alert for each compute that was
not connected, with the compute's name and libvirt's error text. That
text names the host and how it is reached (for example "ssh: connect
to host 10.20.30.132 port 22: No route to host"), while the list hides
host names from users who are not administrators.
- Superusers and users with the global view_instances permission see
each unreachable host and its error, as before.
- Other users see one alert without host details, and only when a host
of their own VMs is down. It explains why those VMs show as unknown.
- The alerts' close button works (data-bs-dismiss was "modal").
Checked against the test setup with an unreachable compute that holds a
user's VM: the user sees the VM as unknown and the alert, but no host
name, address or error text; a superuser still sees them. The tests for
the user fail on the previous code.
The overview page decided whether the host is up with a TCP probe of
the compute's hostname. For an ssh compute named by an ssh_config alias
or reached through ProxyJump, that name and port cannot be reached from
WebVirtCloud directly, so the probe failed while the connection worked:
the page showed the host details and never started the CPU and memory
graphs. The overview now uses Compute.status, the connection
WebVirtCloud uses for everything else.
The compute list shows why a host is not connected (Compute.connection_error)
under "Not Connected", for example a host key verification failure.
Checked against the ssh compute of the test setup through an
ssh_config alias: the previous code renders the overview with the
graphs off, the new one with them on. Both new tests fail on the
previous code.
The instance lists read each VM from libvirt while rendering. When that
failed, the libvirt error replaced the whole page with the error page:
one VM removed out of band, or a host that is down, took the index or
a compute's instances page with it. To avoid that, both index views
left out every VM of a host that was not connected, so users saw their
VMs disappear.
- Instance.info is None when the VM cannot be read. The row then shows
the name and owners from the database, the state "Unknown", and no
title, vCPU, memory or actions.
- Both index views list the VMs of a host that is down. The grouped
view marks the host "Not Connected" and leaves out its CPU and memory.
- A compute's instances page shows the connection error, and no longer
says the hypervisor has no instances when it cannot be reached.
Checked against the test hosts with an unreachable compute added: both
index views list its VM as unknown next to the real VMs. The first
load takes 5.5 s (one bounded connection attempt), the next ones
0.1 s. The new tests fail on the previous code (VMs left out, error
pages).
refresh_instance_database opened the transaction first and then asked
libvirt for the domain list. A host that goes silent on an established
connection keeps that call waiting until keepalive gives up (30 s). On
SQLite, the default database, the open transaction holds a read lock,
so every other write of the app failed meanwhile with "database is
locked", logins (session writes) included. Reproduced in the test
setup: with a relay to a TCP compute frozen after connecting, three
session writes during the 30 s refresh each failed after 5 s.
- The connection check and listAllDomains run before the transaction,
still under the compute lock, so create, delete and migration of the
compute stay excluded as before.
- The transaction only applies the changes. It checks the compute row
first: a compute deleted while libvirt was asked gets no instances
(this ended in a foreign key error before).
A failed listing still leaves the database untouched. With the change,
the same reproduction writes the sessions at once and keeps the
instances. Both new tests fail on the previous code.
The VM rows of the instance lists used instance.proxy.instance.info in
every state check, the vCPU column and the action buttons. A template
calls the method on each lookup, so every row made 7 virDomainGetInfo
RPCs (8 on a compute's instances page), and the memory column read the
domain XML once more.
- Instance.info reads virDomainGetInfo once per request; the lists and
instance_actions.html use it.
- The memory column of both index views comes from the same info
(current memory) instead of the domain XML. The value is the same:
checked against currentMemory for every running and shut-off VM of
the test hosts.
A row now makes one info call and one XML read (the title). The new
test counts the calls in the grouped and non-grouped index and on the
compute instances page; it fails on the previous code with 7 and 8
calls.
libvirt has no timeout for opening a connection. Measured in the test
setup: an unreachable host blocked qemu+tcp for 135 s and qemu+ssh for
133 s; a daemon that accepts and never answers blocks forever. Gunicorn
runs sync workers and the instance list connects to every compute, so
one unreachable compute held a worker for minutes per page load.
- A connection is opened in a helper thread, at most one per
connection at a time. The request waits LIBVIRT_CONNECT_TIMEOUT
(default 5 s). An open nobody waits for any more keeps running; until
it ends, requests get its error at once, and if it succeeds later the
connection is used. A failed open is not repeated for
LIBVIRT_RETRY_AFTER (default 30 s). A stuck host holds one thread and
does not keep other hosts from connecting.
- qemu+ssh runs ssh through vrtManager/libvirt-ssh (the URI's command
parameter), which adds ConnectTimeout=10 and BatchMode=yes, so an
abandoned ssh process ends and no prompt can hang. The user's ssh
configuration (aliases, ProxyJump, keys) still applies.
- The close callback acts only for the connection it was registered
for: a late callback of a replaced connection dropped its successor.
The reason is reported as text. A connection that closed before it
was published is an error.
- Compute.status is a bool (it returned the exception object, so the
"host is down" check in refresh_instance_database never fired); the
message is Compute.connection_error.
Unit tests cover the timeout, retry delay, late success, single open,
a stuck host next to a healthy one and the close callback. Live tests
(vrtManager/test_live_connection.py), for TCP and SSH computes of the
test setup: an unreachable address (no ssh left after 10 s), an endpoint
that never answers, a slow open completing later, a host that goes
silent on an established connection (keepalive ends calls within 45 s,
the next request reconnects), an ssh_config alias and a ProxyJump host.
The OpenStack-style cloud-init endpoints (index, meta_data.json,
user_data) are fetched by booting VMs, which do not log in. Since the
login-required middleware they redirected to the login page, so the
datasource has not worked. They are login_not_required now; they expose
a hostname and the public SSH keys of a VM's owners.
A VM is found by the reverse DNS name of the client address, and VM
names are not unique (only compute and UUID are). user_data merged the
owners of every VM with that name, so a user who named their VM like
another user's VM got their key into that VM at its next cloud-init
run. Keys are now given only when exactly one VM has the name.
The client address is the last X-Forwarded-For entry, which nginx in
the default setup appends; gunicorn listens on 127.0.0.1 only. The bare
except around the reverse lookup is narrowed to OSError.
SHOW_PROFILE_EDIT_PASSWORD was replaced in 2020 by the change_password
permission, settable per user or group. A post_migrate hook applied the
old setting to all users so existing installs kept their behavior, and
asked to remove the setting afterwards; but the settings template still
set it to True. Every migrate, so every upgrade, therefore granted the
permission to all users again and undid an admin's per-user or
per-group choice (with False it would have revoked it from everyone).
- Migration accounts.0008 applies the setting one last time, if a
settings.py still has it; the post_migrate hook is gone.
- New users get the permission when they are created (any path,
including LDAP); the admin user form ticks it and saves the admin's
choice afterwards.
- The template no longer sets SHOW_PROFILE_EDIT_PASSWORD; a check
(accounts.W001) asks to remove it from existing settings, and the
README upgrade notes say so.
The test that removes the permission and runs migrate fails on the
previous code. An upgrade from master with users was simulated: the
setting is applied once, and a permission removed afterwards stays
removed through the next migrate.
computes.views.get_compute_video_models had no URL and no caller, and
ComputeManager.get_video_models was used only by it. The video models
of the create page and the VM settings come from wvmConnect directly.
An optional dhcpd.conf next to the application supplies clone names and
their MAC addresses. get_dhcp_mac_address read it from BASE_DIR, but
guess_clone_name from a fixed /srv/webvirtcloud/dhcpd.conf, so on any
other install path the Guess button found no name while the MAC lookup
still worked. Both use utils.dhcpd_conf() (BASE_DIR) now.
The editor shows its Change button only for a shut-off VM, but
change_xml did not check the state: a page loaded while the VM was off
could still be saved after it was started, and the posted XML then
overwrote every change made to the definition in between. The view now
refuses unless the VM is shut off, with a message.
Creating each option on a RHEL qemu-kvm host showed:
- CD-ROM: the create page offered every disk bus, but the empty CD-ROM
of a new VM works only on ide, sata and scsi. fdc and virtio failed
("Invalid bus type", "does not support ejectable media"), usb failed
with a device named 'Nonea' because vol_dev_type() had no prefix for
usb, and after that libvirt refuses an empty usb disk. Only ide, sata
and scsi are offered now, and another value is refused before any
storage is allocated. vol_dev_type() names usb disks sdX, which also
fixes data volumes on the usb bus.
- USB controller: any input device added a bare <controller
type='usb'/>, which libvirt turns into piix3-uhci (USB 1.1) on q35
instead of its default qemu-xhci. The bare controller is gone; libvirt
adds the machine's default for USB inputs.
- MACs: with more networks than MAC addresses, create_instance indexed
past the list and the request failed with a 500. NICs without a given
MAC now get one from libvirt. MAC addresses are validated (format, not
more than networks) by the page and the API before anything is
created; they went into the XML unchecked.
Unit tests cover the refused values, the MAC rule and the usb prefix.
Live tests create a VM with every offered CD-ROM bus and start it,
check the USB controller with a USB input device and create a VM with
two networks and one MAC.
The form checked both with re.match("^[0-9a-f./:]+|^$"), which tests
only how a value starts: "2001:db8::10/64'/><evil/>" passed and went
into the interface XML, which is built with f-strings, while upper-case
hex was refused. An address without a prefix passed too and then failed
with a ValueError when the XML was built.
The address is parsed with ipaddress.IPv6Interface and stored as
address/prefix (/128 when none is given), the gateway with
IPv6Address. A static configuration needs an address, and the XML gets
a route only when a gateway is given instead of gateway=''. The HTML
patterns accept upper-case hex.
Eight of the ten form and XML checks fail on the previous code.
create_instance set secure='yes' (and with it SMM) only while switching
a non-q35 machine to q35. With q35 already chosen, a secure boot loader
was defined with secure='no' and without SMM. Secure boot is now
requested whenever the loader is a secure boot build; the switch to q35
is unchanged.
libvirt 11 on RHEL fills in secure='yes' and SMM from the firmware
descriptor of a known loader, so there the domain was already correct;
without such a descriptor (older libvirt, custom loader) it was not.
A unit test checks what WebVirtCloud requests (fails on the previous
code); a live test creates a q35 VM with the host's secure boot loader
and starts it.
create_instance passed the loader list where find_uefi_path_for_arch
expects the machine type. The domain capabilities were then read for
the default machine (pc) instead of the chosen one: on a RHEL host a q35
VM was offered /usr/share/OVMF/OVMF_CODE.secboot.fd, which q35 does not
list, instead of /usr/share/edk2/ovmf/OVMF_CODE.fd. The machine type is
passed now.
The page also read dom_caps["loader_enums"] into an unused variable;
on a host without loader support the key is missing and the page failed
with a KeyError. The line is removed.
A unit test covers the call and the host without loader support (it
fails on the previous code). A live test creates a q35 VM with the
offered UEFI firmware, checks that libvirt lists that loader for q35 and
starts the VM; it fails on the previous code.
attach_disk (new or existing volume, CD-ROM), add_guest_agent and
remove_guest_agent acted only when the status was running (1) or shut
off (5). On a paused VM they changed neither definition and the request
reported success. They now use the same rule as the NIC edits: the
running definition while the VM is active, paused included, and the
persistent one always. remove_guest_agent reads each definition from its
own XML, so a channel present only in the persistent one is removed too.
Unit tests check paused, running and stopped VMs (paused fails on the
previous code). The live effects test adds attach_disk and
add_guest_agent, and now also checks that edits meant for the running VM
reach it, which showed that an ejected CD-ROM keeps an empty <source>
in the live XML; NIC unplug is left to the guest tests, as these VMs
have no OS to acknowledge it.
New live guest tests compare what the guest agent reports (online vCPUs,
memory blocks, disk serials and buses, NIC MACs) with the persistent
definition, through the web views:
- at boot, with a data disk;
- after force off, CPU and memory resize and power on (2 vCPUs, 3 GiB);
- a hotplugged disk at once; a bus and serial change after powercycle;
- a NIC MAC change after powercycle, not before;
- a clone boots with its own MAC and disk copy;
- reverting a running snapshot removes a NIC added after it, in the
guest and in the definition, and leaves the VM running;
- a suspended guest does not answer and answers again after resume.
Every request in these tests must redirect (302), so a failing view is
not mistaken for a no-op. Only NICs with QEMU's 52:54: prefix are
compared, as a guest may create its own interfaces. All 13 guest tests
pass against a Rocky Linux 9 guest.
The live tests so far ran VMs without an OS, so everything that needs the
guest's cooperation was untested. With TEST_LIBVIRT_GUEST_IMAGE set to an
installed Linux image with qemu-guest-agent, instances/test_live_guest.py
boots wvc-test VMs from qcow2 overlays of it (the image is only read) and
checks through the web views:
- delete_vol on a running VM: the guest releases the disk, so it leaves
the live and persistent definitions and the volume is deleted;
- detach_vol removes the disk from the running guest and keeps the volume;
- add_network/delete_network: the guest sees the hotplugged NIC, and the
unplug completes in both definitions;
- set_vcpu brings a hotplugged vCPU online in the guest;
- osinfo returns the guest agent's OS, time zone and host name, and the
agent reports the VM's NIC by MAC;
- poweroff shuts the guest down through libvirt.
The image must be a BIOS-bootable qcow2 volume in a storage pool other
than wvc-test; the tests are skipped while a running VM uses it. They pass
against a Rocky Linux 9 guest on a RHEL qemu-kvm host; the guest had no
address on the NAT network (its network configuration is static), so only
the agent's NIC report is checked, not an address.
The web app (wsgi.py), novncd and socketiod set SIGCHLD to SIG_IGN at
import. The kernel then reaps every child at once, so subprocess loses
the exit status and a failing command reports 0, in the server and in
anything else run in that process: the test suite had to reset the
signal in two tests.
It was added in 2023 against one defunct bash per gunicorn worker and
per novncd. That bash came from the settings template of the time,
which ran .dec_ldap_pwd.sh with Popen and never waited for it; the
template stopped doing that in 561fedf. A settings.py still holding
that line leaves at most one zombie per process, not a growing number.
- wsgi.py, novncd, socketiod: the global SIG_IGN is removed. subprocess
reaps its own children, websockify installs its own SIGCHLD reaper
for novncd's connection processes, and novncd's SSH tunnels wait for
their ssh process.
- socketiod: the pty child execs consolecallback instead of running it
and then returning into the server's code; disconnect reaps that
child (waitpid) instead of any child (wait).
- The settings template drops its unused subprocess import; the two
test workarounds are removed.
New tests load the web app and novncd in a fresh interpreter and check
that SIGCHLD is untouched and a failing command still returns 1; both
fail on the previous code. A patched novncd in the test container
served five connections and left no child behind.
Web views:
- 34 views returned to request.META["HTTP_REFERER"] + "#tab": without
a Referer they failed with a TypeError (500), and a Referer from
another site was followed (open redirect). They return through
_back(), which keeps a same-site Referer and otherwise opens the VM
page at the tab. get_safe_redirect uses the same check. Rejected disk
options also return to the VM's disks tab instead of the VM list.
- migrate_instance returned silently when the target was the current
compute or unreachable, so the web view logged "Instance is migrated"
and the API answered "instance migrate is started". It now raises;
the view reports the error and logs only a completed migration, and
an unknown compute id is a 404 instead of a 500. The automatic
migration after a clone skips a target that is the current compute.
- add_owner and del_owner failed with a 500 without their ids, add_owner
also when the user was already an owner, and del_owner deleted the
given ownership record even when it belonged to another VM. Ids are
validated, a record of another VM is a 404, and a repeated owner is
reported.
- Broad handlers that hid programming errors are narrowed: clone catches
libvirt, value and OS errors (a lock timeout included), ownership
lookups catch DoesNotExist. The DRBD status no longer needs its bare
except: the VM name is escaped in the regex, which was the only thing
that could fail there.
API:
- A libvirt error under /api/ is JSON with status 400, and a lock
timeout 409, instead of the HTML error page. This lives in the
existing ExceptionMiddleware, so existing settings need no change.
- Volume create/delete on an inactive pool answer 409, and invalid
volume data 400 with the serializer errors, instead of 200.
The new tests cover every view that returns with _back() without a
Referer and with a foreign one, the migrate failures on web and API,
the owner ids, the API error format and the volume statuses; nine of
the ten fail on the previous code.
Branches no longer need a pull request to master to be tested; a
branch is checked when it is pushed and lands on master by
fast-forward once it is green. Pull requests to master still run the
checks. CodeQL keeps running on master, its pull requests and its
weekly schedule.
Comments, docstrings and live test names cited item IDs of a roadmap
that is not part of the repository. They now say what is meant, or
nothing where the sentence stands without them. Test assertions and
behavior are unchanged.
- host_is_up returned the exception object for a down host, which is
truthy: check_user_quota then connected to it and failed, so a user
with a VM on a down host could not open its page, clone or resize,
and migrate_instance's "target is down" check never fired. It returns
True or False now, closes its socket, and accepts IPv6 and host:port.
- The connection manager no longer holds its global lock while a new
connection is opened, so one slow host does not delay new connections
to the others.
Bounding the connection time itself (including SSH aliases and jump
hosts) is left for a separate change.
The sync deleted the Instance of every VM missing from its compute, and
its owners and rights went with it. A VM moved with virsh, or undefined
and defined again, came back with no owner.
Now, when a VM disappears from a compute:
- if the same UUID is recorded on another compute, it moved: its owners
and template flag go there;
- otherwise they are kept in an InstanceTombstone and restored when the
UUID appears again within INSTANCE_OWNERSHIP_RETENTION_DAYS (30).
A UUID recorded on two computes at the same time is a copy and gets no
owners. VMs deleted through WebVirtCloud leave no tombstone.
Web form: the cache mode and the UEFI firmware value were checked only
after the disk volume had been created; an invalid one answered 500 and
left the volume. All checks now run before any allocation. Volumes the
request creates are removed on any failure up to the definition; ones
that cannot be removed are reported. After the definition the VM uses
them, so a later failure keeps them.
API: the new volume was never removed when the definition failed, and
the error was a 500. It is removed now and a libvirt error returns 400.
- change_network detached the NIC and attached one built from scratch,
dropping its QoS, link state and other settings. It now edits the NIC
of the persistent definition in place and defines once; a running VM
gets the change at its next start, as the UI already says.
- add_network, delete_network and set_link_state did nothing on a
paused VM, and set_link_state rewrote the persistent NIC from the live
XML. They now update the running definition when the VM is active and
the persistent one always, each from its own XML.
- Boot and console edits used xml.etree, which renamed the namespace
prefixes of other tools' metadata (<app:info> became <ns0:info>); they
use lxml now.
- The NIC table and the clone form are filled from the persistent
definition (Instance.config_networks, config_disks), so a pending NIC
change is shown and can be edited again.
A live test runs 18 domain edits on stopped, running and paused VMs
whose definition has elements WebVirtCloud does not manage, and checks
for each the effect, that nothing else in the persistent definition
changed, and that persistent-only edits leave the running VM alone.
Clone:
- only shut-off VMs are cloned, as the UI already required: a running
guest keeps writing while its disks are copied; the definition comes
from the persistent XML;
- MAC addresses and disk destinations are checked before anything is
allocated;
- every volume the clone creates (NVRAM and disks) is tracked; if a
later copy or the definition fails, only those are deleted, and any
that cannot be deleted are named in the error. Before, a failed clone
left its earlier copies behind.
libvirtError(msg) reports the thread's last libvirt error instead of
msg when there is one, so WebVirtCloud's own errors could show an
earlier, unrelated failure. util.OperationError, a libvirtError
subclass that keeps its message, replaces those raises in vrtManager,
instances and nwfilters; 'except libvirtError' still catches it.
SPICE was removed earlier; its leftovers and the console type choice go
too:
- the console settings form, update_console and the backend no longer
offer or change the type; an invalid console form now shows its
errors instead of being ignored;
- the create form, the API serializer and create_instance no longer take
a graphics type, new VMs get VNC; get_graphics_types and the graphics
entries of the domain capabilities are gone;
- migration appsettings.0013 deletes QEMU_CONSOLE_DEFAULT_TYPE;
- a VM defined elsewhere with another graphics type gets "Console type
... is not supported".
The locale catalogs still reference the deleted SPICE templates in
source comments; regenerating them is left to the translation work.
The old code edited the live XML, called attachDevice and redefined the
domain from it: a stopped VM lost its VNC password, a paused VM crashed
with UnboundLocalError, an unknown CD-ROM was silently ignored (on a
running VM the last disk of the loop could be updated instead), and a
missing ISO raised AttributeError.
Media changes now use updateDeviceFlags on the requested CD-ROM, built
from the live and the persistent definition separately. A CD-ROM
missing from one definition, an unknown ISO, an ISO name present in
more than one pool, or a change that reached only the running VM is
reported to the user. The mount and unmount log messages were swapped.
Edits that redefine a domain read the wrong XML:
- boot menu/order, vCPU hotplug and QoS read XMLDesc(0), which has no
secrets, so every such edit, even on a stopped VM, removed the VNC
password from the definition;
- console, video, CPU, memory and title/description edits read the live
XML, so on a running VM pending changes (e.g. a new memory size) were
reverted and runtime state was written into the definition.
All of them now read VIR_DOMAIN_XML_INACTIVE | SECURE (PERSISTENT_XML).
The console setters take the first graphics device of that definition
instead of looking it up by the live type.
Disk resize no longer redefines the domain. A running VM's image files
grow through blockResize, since qemu-img cannot take the lock QEMU
holds; block volumes (e.g. LVM) are grown by the storage pool first and
QEMU is then told the new size.
Live tests reproduce the losses on a RHEL qemu-kvm host; unit tests
check the XML flags, the console setters and the resize paths in CI.
