A VM that changes its memory faster than the network carries it does
not finish a live migration without auto converge (measured: a guest
rewriting about 1.2 GB/s did not finish in 90 s over an 8.5 Gbit/s
link, and finished in 16-18 s with it). Auto converge only slows the
VM's CPUs while the migration cannot keep up, so a calm VM is not
affected.
The migrate form ticks it for a running VM. The API already documented
it as on by default but read a left-out option as off; the serializer
now has the defaults (auto converge on, compress and unsafe off) for
JSON and form-encoded requests, and the schema shows them.
A VM that changes its memory faster than the network carries it never
finishes a live migration without auto converge: measured with a guest
rewriting about 1.2 GB/s over an 8.5 Gbit/s link. The migration then
ran until gunicorn ended the request after 600 s. libvirt cancelled it
when the request went away and the VM stayed on the source, but the
user got an error page and nothing was logged.
The app now cancels a migration that has not finished after 540 s
(MIGRATION_TIME_LIMIT, below the 600 s of gunicorn.conf.py; a test
keeps it below). The failure is reported and logged like any other:
the migration was cancelled, the VM stays on the source host, and a
VM like that needs auto converge. A timer that fires as the migration
ends finishes before the compute locks are released, and an error
the cancel did not cause keeps its own message.
With "VM Clone Auto Migrate" on, a new clone was migrated to a random
compute, whatever storage, CPU or network that host had, and a failed
migration showed as a failed clone although the clone existed.
The setting and its code are removed; migration appsettings.0015
deletes the setting. A clone can be migrated afterwards like any VM.
The README's update notes list the migration changes: the migration
address, storage pools for the disk check, and this setting.
What migrate_instance did around the migration itself:
- It checked the destination with a 1 s TCP probe of the libvirt port,
which passes while libvirt is down or refuses the login. It now uses
the compute's connection status, as the rest of the app does.
- Two requests for the same VM waited for the compute locks one after
the other; the second then migrated with the VM's old location. The
VM's record is read again under the locks, and the request is
refused when the VM is no longer on the source.
- Restoring autostart on the destination came after the migration but
before the database move; when it failed, the VM ran on the
destination while the app recorded it on the source. Autostart is
now read from libvirt and a failure to restore it is only logged.
- A database failure after a successful migration was logged as
nothing; the log now says the VM migrated and recording it failed.
- When the destination's list had already made a record for the VM,
the merge kept the owners but dropped the template mark.
Migration needs the VM's disks on storage both hosts reach, and nothing
checked that. A VM with a disk the destination did not have failed
during a live migration; a shut-off VM's definition moved and could not
start there.
moveto now refuses the migration before anything changes, naming each
disk (target and source) that is not a volume of the destination's
active storage pools. Checked: the disks, CD-ROM images and block
devices of the live and the persistent definition, pool volumes, and
backing files, as the definition or the destination's volume lists
them (a shut-off VM's definition does not). Pools are refreshed before
a disk counts as missing, so a file another host made on shared
storage is found. Network disks and network backing images are
reached the same way from every host and are not checked; disk types
that cannot be checked (dir, nvme, ...) are refused.
This checks that the destination knows a volume, not that it is the
same file: shared storage should be a storage pool on each host. A
shared file outside any pool, or a block device no pool covers, is
refused.
moveto read the VM state for each flag and dropped options that did
not fit it, so a request could run as something else than asked; the
log then named the mode from the form ("live" or "offline"), not the
one used. The API did not log at all and answered "instance migrate is
started" after the migration had finished.
- moveto reads the state once. A running VM migrates live, or without
live paused for the transfer (non-live); a paused VM stays paused; a
shut-off VM moves offline. Live for a shut-off VM, offline for a
running or paused one, and any other state are refused before
anything changes. unsafe, auto converge (live only) and compressed
apply to a running VM.
- migrate_instance logs the outcome for the VM page and the API: the
mode and options used, or the error. The API answers
"instance is migrated (<mode>)" and accepts requests that leave out
an optional option.
- Post copy is gone from the form, the view and the API: only the
flag was set, and without migrateStartPostCopy, which the app never
called, the migration never switched to post copy.
libvirt's offline migration fails on some builds with "operation
failed: domain is no longer running" (RHEL-156800: Rocky/RHEL 9.8
libvirt 11.10.0-12 has the change that broke it, not the fix, which is
upstream in 12.2.0). A shut-off VM could not be migrated at all there.
A shut-off VM is only its definition, as the disks are on storage both
hosts share, so moveto now moves that itself:
- refused first when the VM has snapshots or a saved state, or the
destination already has a VM with its UUID or name;
- defined on the destination from the source's migratable XML, then
undefined on the source; NVRAM and TPM state stay on the source;
- after a failure the VM stays defined on one host: the destination
definition is removed only while the source still has its own, and
a VM another client started on the source meanwhile gets its
definition and autostart back there.
The destination's qemu migration hook does not run for it. Running and
paused VMs still migrate through libvirt.
Migration is client-driven: libvirt sends the VM's memory to the
destination's own host name unless a migration URI is given, and the
app never gave one. Where the hosts cannot resolve each other's names
(or the names point at a management network) every migration failed
with "Unable to resolve address '<host name>' service '49152'".
A compute now has an optional migration address (an IP address or a
host name, no scheme or port), editable in the compute form and the
API. A VM migrating to that compute is sent to tcp://<address>
(bracketed for IPv6). Empty, the default for existing computes, keeps
the previous behavior. The compute's host name is not reused, because
for SSH computes it can be an ssh_config alias.
The migration log entry names the address when one is set.
Migration left the VM defined on the source host unless "Delete
original" was ticked (VIR_MIGRATE_UNDEFINE_SOURCE). With the shared
storage migration needs, that definition points at the disks the
destination VM is running on: the VM showed on both hosts and could be
started (or autostarted) a second time against the same disks; only a
configured lock manager stops that.
moveto now always passes VIR_MIGRATE_UNDEFINE_SOURCE with
VIR_MIGRATE_PERSIST_DEST. The option is gone from the form, the view,
the API (MigrateSerializer.xml_del) and the clone auto-migration. It
removes the definition, not the disk files.
The TCP and TLS compute forms never show the stored password, so the
field is empty on every edit. Saving the form without typing the
password again replaced it with nothing, and the next connection to the
host failed to authenticate.
An empty password field on an edit now keeps the stored password; a
new value replaces it.
The VM page polls the VM state every 5 s and reloads when it differs
from the state the page was rendered with. Since the polling moved to
pollWhileVisible, that state was kept in a top-level "var status",
which is not a page variable but the browser's window.status. Where the
browser does not keep a value there, the poll compared the answer with
"" and reloaded the page every 5 s, back to its first tab.
The variable is now vm_status. A Node test runs the page's script with
a window.status that keeps nothing.
Nothing limited failed logins: a password could be guessed at the speed
the server answers. django-axes (8.3.1, Django 5.2, Python 3.10-3.13)
now counts them.
- A user is locked for the address the failed logins come from
(username and IP together), so an attacker cannot lock someone out
from elsewhere and users behind one NAT address do not share a lock.
A successful login resets the count. axes sits in front of every
authentication: the login page with or without OTP, LDAP and the API.
- The number of failed logins (default 5) and the lock time in minutes
(default 15) are on the Settings page (LOGIN_FAILURE_LIMIT,
LOGIN_LOCKOUT_MINUTES, added by migration appsettings.0014) and apply
at once; an invalid value falls back to the default. Unlock with
"manage.py axes_reset_username <username>".
- Behind the bundled nginx every request comes from 127.0.0.1; the
client address is read from X-Real-IP, which nginx sets, but only for
requests from a loopback address or from LOGIN_TRUSTED_PROXIES, so a
forged header does not escape a lock.
settings.py.template gets the app, middleware, backend and AXES_*
settings; README tells existing installations what to copy. Tests that
call client.login now pass a request, which the axes backend needs.
The new tests fail with axes disabled.
The first page of the create wizard built the next page's URL from a
reversed ".../x86_64/pc/" by replacing "x86_64" with the chosen
architecture and then the first "pc" with the machine type. An
architecture containing "pc" got the machine type inside its name:
ppc64le with pseries went to /instances/create/1/ppseries64le/pc/, so
VMs could not be created on PowerPC hosts.
The URL is now reversed with the placeholders __arch__ and __machine__,
which the script replaces with the URL-encoded values.
The test renders the page, runs the script with node and fails on the
previous template for ppc64le.
When a VM action failed because its host could not be reached (the
connection could not be opened, or it dropped during the call), the
error page showed libvirt's connection error to any user: it names the
host and how it is reached, e.g. "ssh: connect to host 10.x port 22".
The lists and the VM page already keep that to administrators.
- Opening a connection now fails with ConnectionFailed (a subclass of
OperationError). A dropped connection is recognized by its error
domain and code (RPC/remote/SSH layer with a transport code, measured:
a keepalive timeout is VIR_FROM_RPC/VIR_ERR_INTERNAL_ERROR); an
operation the daemon does not support (VIR_ERR_NO_SUPPORT) is not
treated as one.
- ExceptionMiddleware (pages and API) and the views that report the
error themselves (snapshot, revert, disk deletion, clone) show it to
superusers and view_instances users, and to everyone else "The host
of this virtual machine cannot be reached right now." Errors of the
operation itself ("domain is not running") are shown as before.
Checked on the test setup: powering on a VM of an unreachable compute
shows the owner the generic text and the administrator the error. The
tests fail on the previous code.
Four lookups built XPath expressions from values that come from
requests (CodeQL py/xpath-injection, plus get_capabilities, which
CodeQL did not flag; the pages are for superusers):
- The architecture in the create wizard's URLs went into
arch[@name='...'] in get_emulator, get_machine_types and
get_capabilities: "nope' or '1'='1" matched every architecture. It is
now an XPath variable ($arch), which is never parsed as XPath.
- Network filter rules were found with a predicate built from the
posted action, direction and priority: a quote raised SyntaxError (an
error page). The rule is now found by comparing its attributes.
- unset_qos put the posted direction into the path, so "*" removed
every bandwidth setting. It accepts inbound and outbound, as set_qos
does; the network page shows an error for anything else.
No other XPath in vrtManager is built from a value; the remaining
expressions are constants. The tests fail on the previous code.
The settings page lists the themes of SASS_DIR and writes wvc-main.scss
into it when the theme changes; the value could be any path, so a
superuser could make the app list and write files anywhere it may
(CodeQL py/path-injection).
SASS_DIR is now accepted only if it resolves to an existing directory
inside the project (the default is dev/scss). A stored value outside it
is not used: the page shows an error and offers no themes, so nothing
is written. Installations that keep their SCSS outside the project
need to move it under it.
The tests fail on the previous code for an absolute path, "../",
a path climbing out and a missing directory, and for a stored value
outside the project.
The settings page took the posted theme name as it came and wrote it
into the @import lines of wvc-main.scss and the compiled CSS: a name
with "../" or a quote imported files outside the themes directory
(CodeQL py/path-injection; the page is for superusers).
The name must now be one of the directories in SASS_DIR/wvc-themes, the
list the page already offers; anything else is refused with a message
and nothing is written. themes_list is empty when SASS_DIR has no
themes, so every name is refused then.
Checked with the real dev/scss: flatly is applied and compiled as
before, "../../etc" is refused and leaves the CSS alone. The test fails
on the previous code.
The test and lint workflows set no permissions, so their GITHUB_TOKEN
had the repository's default access (CodeQL
actions/missing-workflow-permissions). Both only check out and build
the code: they now ask for contents: read.
.github/workflows/codeql-analysis.yml was disabled on GitHub; code
scanning runs through GitHub's default CodeQL setup instead. The file
only kept the old analysis category (and its 2023 alerts) around.
The expression that checks a storage pool's target directory nested a
repetition inside another, so a crafted target backtracked
exponentially: "," followed by 22 dashes and "!" took 1.3 s, and every
two more characters four times as long. The field allows 100
characters, so one form post could hold a worker for good (CodeQL
py/redos; the page is for superusers).
The new expression has no nested repetition and accepts exactly the
same targets: compared with the old one on 200,000 random strings over
letters, digits, "/", ".", "-", "_", ",", "!", space and newline, they
agree on every string. A crafted 100-character target is now checked
in microseconds. The tests pin the accepted and refused targets and
time the crafted one.
The user edit page redirected to its ?next= parameter after saving,
whatever it held: a link to the edit page with next=https://other.site
sent the administrator who saved the form to that site (CodeQL
py/url-redirection). The account page links there with a next of this
site.
next is now followed only when url_has_allowed_host_and_scheme accepts
it for this host, as user_instance_delete already does; otherwise the
user list. The test fails on the previous code for an absolute URL, a
scheme-relative one and a backslash variant.
Templates do not count towards a user's quota and cannot be started.
Staff owners may mark and unmark their VMs as templates, so a staff
user could mark a VM as a template, use the freed quota elsewhere, and
unmark it again: the VM came back without any check, over the quota.
For users with a quota (everyone but superusers), turning a template
back into a VM now passes check_user_quota with the VM's size (one VM,
its vCPUs, memory and disks). If the quota is reached or cannot be
verified, it stays a template and the user is told why. Marking a VM as
a template frees quota and is not checked. The view takes the
per-user quota lock like the other quota-checked changes.
The existing template-flag test gives its mocked VM a size. The new
tests fail on the previous code.
The VM page's getters read the domain XML on their own: one load made
86 libvirt calls for a real VM, 29 of them for the same three XML
documents (live, live with secrets, persistent) and 6 for the host
capabilities. Each call is a round trip to the host.
wvmInstance.cached_reads() is a block in which _XMLDesc (per flags
value) and get_cap_xml are read from libvirt once; it ends with the
block, even on an error, does not nest, and _defineXML drops it.
Outside a block every read goes to libvirt as before. Only the VM page
view, which does not change the VM, uses it, around its rendering.
Views that change the VM keep reading fresh XML: most changes (devices,
QoS, memory) do not go through _defineXML, and several views read,
change and define more than once in one request.
Measured on the test host: 86 -> 55 calls (XMLDesc 29 -> 3,
getCapabilities 6 -> 1) for one VM, 82 -> 54 for another; the page took
0.23-0.25 s before, 0.14-0.15 s after. The HTML of both pages is
identical before and after (CSRF tokens aside). A test renders the page
on libvirt's test driver through the real getters: 25 XML reads before,
one per kind now.
django-bootstrap-icons serves icons from static/icon_cache and fetches a
missing one from the CDN while the page renders; a server without
internet access shows an error text instead of the icon.
- copy (Clone on the instance lists), envelope-open ("Lost OTP?" page)
and save (VM settings, network page) were not cached; they are now,
fetched with django-bootstrap-icons itself.
- The user and group lists asked for 'exclamation-triangle ' with a
trailing space, an icon that does not exist; the warning icon never
showed.
A test checks that every {% bs_icon %} in the templates has its cache
file; it fails on the previous files.
/accounts/email_otp/ mails a user their OTP QR code and needs no login.
Measured with the default settings (OTP disabled):
- It was reachable although OTP was off, and created OTP devices.
- Every request sent a mail: 20 requests for one address, 20 mails, so
anyone who knew an address could fill that mailbox.
- An address shared by two users, or a mail server that could not be
reached, ended in an error page (500).
- Inactive users were mailed too.
The page now exists only while OTP_ENABLED is set. It mails a single
active user whose address matches (case-insensitively), at most once
every 5 minutes per user (Django's cache: with several workers the
bound is per worker), and logs a failed delivery. The answer is the
same in every case, so it does not reveal whether an address belongs
to a user.
The QR codes come from django-qr-code (segno); both the mail and the
account page render them without the qrcode package removed earlier.
The new tests fail on the previous code.
The instance lists, the VM page header and the power tab knew three
libvirt states: running, paused and shut off. A VM that was blocked,
shutting down, crashed, suspended by its guest or without a state got
an empty status cell and no actions, so a crashed VM could not even be
forced off from the list. The lists also offered actions users could
not use: Suspend and Resume (views for superusers) to everyone, and
power actions to users who see every VM through the read-only
view_instances permission.
instances/states.py holds the rules in one place: the label and color
of each state, the actions each state allows, and which of them a user
may use (superusers all; a VM's owners all but suspend and resume, and
nothing on a VM an administrator suspended, as the VM page already
showed; nobody else any). Blocked is a running VM; shutting down,
crashed and suspended by the guest get Force Off, after which the VM
can be started. A shut-off template offers Clone to whoever has the
clone permission, as the clone view allows.
The templates read them through two tags (vm_states): the lists and the
VM page header show the state, instance_actions.html has one slot per
button, each enabled or disabled from the allowed actions. The power
tab offers Force Off in the states that need it.
The new tests check the rules directly and render every state in the
lists and on the VM page for a superuser, an owner and a
view_instances user; they fail on the previous code.
The admin page that creates a user saved the form first, which wrote
the user with the raw password in the password column, and hashed it
with a second save afterwards. The raw password was in the database,
and in its journal and any copy taken meanwhile, and stayed there if
anything failed between the two writes.
- UserCreateForm hashes the password before the first save; groups
and permissions are saved as before.
- The form applies AUTH_PASSWORD_VALIDATORS, after the user fields are
set (so the similarity check sees the new username), as the password
change forms already did. The settings template configures no
validators, so nothing changes unless an installation sets them.
The tests record the SQL parameters of the request: the previous code
wrote the raw password in the INSERT; it also accepted passwords that
configured validators reject.
The VM page polls the VM state every 5 s and, while its Stats tab is
open, the statistics every 10 s; the compute overview polls the host
graph every 5 s. All used setInterval:
- A new request went out on schedule while the previous one was still
open. When a host goes silent, a stats request waits up to 30-45 s,
so each open page held several sync workers at once.
- Each time the Stats tab was shown, the page made new charts and a
new interval without stopping the old ones, and polling went on on
the other tabs of the page.
- Hidden pages kept polling at full rate.
Measured in a browser against a test server (old code / new): Stats tab
shown 4 times, 31 s on it: 12 / 2 requests; 31 s on another tab of the
page: 12 / 0; page hidden for 31 s: 15 / 0, and the new code asks once
right after the page is shown again; with 25 s answers, at most 15 / 1
requests open at once.
static/js/poll.js repeats a request while the page is visible: the
next one 5 or 10 s after the previous answer, none while hidden, one at
once when shown again. The Stats tab makes its charts once and polls
only while shown. The answer handler runs after the poller's own
bookkeeping, so an error in it (a chart for a disk added meanwhile)
does not stop the polling. A Node test of poll.js runs with the test
suite when node is installed.
Each open VM page asks for its statistics every 10 seconds. For a
running VM the stats view measured the CPU, then each disk, then each
NIC, sleeping one second for each. With two disks and a NIC a request
held a sync gunicorn worker for over 4 s, so a few dozen open VM pages
could keep every worker busy.
wvmInstance.usage() replaces cpu_usage, disk_usage and net_usage, the
stats view's only callers: it reads all counters, sleeps one second
once and reads them again. Values and units are unchanged (CPU percent
of the host, disk bytes, network bits per second); compared against the
previous methods on the same counters, the results are identical.
Measured on the test host for a running VM with two disks and a NIC:
the stats request took 4.1-4.3 s before, 1.1-1.3 s now. The new test
fails on the previous code.
A VM owner who may manage the console (is_change and is_vnc) could set
its listen address to all interfaces and remove its password: a VNC
console reachable from the network without a password. Checked on a
test host: the owner's form changed the VM's listen address to 0.0.0.0
and then removed the password.
- The listen address field is disabled for users who are not
superusers; a disabled field ignores posted values. They can still
set or change the password and the keymap.
- A console whose persistent definition does not bind it to loopback
(or a Unix socket) only loses its password through an administrator.
A binding the XML does not pin down counts as not local: listen type
network, or no address (the host's default in qemu.conf).
On the test host, the owner's form now leaves the listen address alone
and is refused when it removes the password of a console listening on
0.0.0.0; with the console on localhost the password can be removed.
The new tests fail on the previous code.
The connection manager registered libvirt's event implementation and
started the event loop thread when vrtManager.connection was imported.
novncd imports it and then forks a child per console (websockify's
multiprocessing), so every child was forked from a process with a
running libvirt thread, which libvirt does not support, and the child
had no loop thread of its own: keepalive never ran there. Measured in
the test setup: in such a child, a call on a host that went silent
still waited after 70 s.
- The loop starts with the first get_connection of a process. The
state is per process and shared by all managers; libvirt's event
implementation is registered once and a child inherits it.
- After a fork, the child renews the lock and starts its own loop on
first use. It stops using the inherited connections, which are the
parent's sockets, but keeps them: releasing them would run libvirt
destructors in the child. No process connects before forking today
(novncd's parent does not connect, gunicorn does not preload); this
keeps such a case safe.
With the change, novncd's parent has no libvirt thread, the console
child starts its own, and the same call fails after 29 s with a
keepalive timeout. A test forks a real child; it fails without the
fork handling.
The VM page reads the VM from libvirt throughout. When its host could
not be reached, the connection error replaced the page with the error
page and showed libvirt's message, which names the host and how it is
reached, to any user who may view the VM.
The page now checks Compute.status first. If the host is not connected,
it goes to the instance list with a message: administrators (superuser
or view_instances) get the host and the error, other users only that
the VM's host cannot be reached right now. The list shows the VM as
unknown.
This covers the page itself; actions on the VM still report libvirt's
error if the host drops. The detail page tests now mock a connected
host.
Checked against the test hosts: a real VM's page opens as before, a VM
on an unreachable compute leads to the list with the user's or the
administrator's message. The new tests fail on the previous code.
check_user_quota added up the vCPUs, memory and disks of the user's
VMs, but skipped a VM whose host failed a TCP probe. A host that was
down therefore let users exceed their quota, and the probe always failed
for ssh computes named by an ssh_config alias or reached through
ProxyJump, so their VMs were never counted. A VM that could not be read
turned the page into the error page instead.
- Hosts are checked with Compute.status, the connection WebVirtCloud
uses, once per host. A VM that cannot be read (host down, VM gone)
makes the usage unknown: clone and resize up are refused with their
own message (fail closed).
- A request that adds nothing is not checked, so a user over quota can
still shrink a VM. Users without limits, and an exceeded instance
count (from the database), need no host either.
- Memory resize of a running VM changes only the current memory, but
the quota was charged with the submitted maximum. It is now charged
with no change, and resize_mem gets the current maximum, so a VM that
shuts down meanwhile cannot take an unchecked one.
- The VM page no longer computes the quota on every load (it read every
VM of the user on every host for a "quota reached" banner). The limit
is reported when a change is refused.
The test that expected a down host to pass the quota now expects it to
be refused. Checked against the test hosts: with a VM behind an
ssh_config alias the previous code counted 29 vCPUs, the new one 31;
with a VM on an unreachable host the previous code allowed +1 vCPU,
the new one refuses it after 5.1 s and allows -1 at once. The new
tests fail on the previous code, except the ones guarding unchanged
behaviour.
The instance list showed every user an alert for each compute that was
not connected, with the compute's name and libvirt's error text. That
text names the host and how it is reached (for example "ssh: connect
to host 10.20.30.132 port 22: No route to host"), while the list hides
host names from users who are not administrators.
- Superusers and users with the global view_instances permission see
each unreachable host and its error, as before.
- Other users see one alert without host details, and only when a host
of their own VMs is down. It explains why those VMs show as unknown.
- The alerts' close button works (data-bs-dismiss was "modal").
Checked against the test setup with an unreachable compute that holds a
user's VM: the user sees the VM as unknown and the alert, but no host
name, address or error text; a superuser still sees them. The tests for
the user fail on the previous code.
The overview page decided whether the host is up with a TCP probe of
the compute's hostname. For an ssh compute named by an ssh_config alias
or reached through ProxyJump, that name and port cannot be reached from
WebVirtCloud directly, so the probe failed while the connection worked:
the page showed the host details and never started the CPU and memory
graphs. The overview now uses Compute.status, the connection
WebVirtCloud uses for everything else.
The compute list shows why a host is not connected (Compute.connection_error)
under "Not Connected", for example a host key verification failure.
Checked against the ssh compute of the test setup through an
ssh_config alias: the previous code renders the overview with the
graphs off, the new one with them on. Both new tests fail on the
previous code.
The instance lists read each VM from libvirt while rendering. When that
failed, the libvirt error replaced the whole page with the error page:
one VM removed out of band, or a host that is down, took the index or
a compute's instances page with it. To avoid that, both index views
left out every VM of a host that was not connected, so users saw their
VMs disappear.
- Instance.info is None when the VM cannot be read. The row then shows
the name and owners from the database, the state "Unknown", and no
title, vCPU, memory or actions.
- Both index views list the VMs of a host that is down. The grouped
view marks the host "Not Connected" and leaves out its CPU and memory.
- A compute's instances page shows the connection error, and no longer
says the hypervisor has no instances when it cannot be reached.
Checked against the test hosts with an unreachable compute added: both
index views list its VM as unknown next to the real VMs. The first
load takes 5.5 s (one bounded connection attempt), the next ones
0.1 s. The new tests fail on the previous code (VMs left out, error
pages).
refresh_instance_database opened the transaction first and then asked
libvirt for the domain list. A host that goes silent on an established
connection keeps that call waiting until keepalive gives up (30 s). On
SQLite, the default database, the open transaction holds a read lock,
so every other write of the app failed meanwhile with "database is
locked", logins (session writes) included. Reproduced in the test
setup: with a relay to a TCP compute frozen after connecting, three
session writes during the 30 s refresh each failed after 5 s.
- The connection check and listAllDomains run before the transaction,
still under the compute lock, so create, delete and migration of the
compute stay excluded as before.
- The transaction only applies the changes. It checks the compute row
first: a compute deleted while libvirt was asked gets no instances
(this ended in a foreign key error before).
A failed listing still leaves the database untouched. With the change,
the same reproduction writes the sessions at once and keeps the
instances. Both new tests fail on the previous code.
The VM rows of the instance lists used instance.proxy.instance.info in
every state check, the vCPU column and the action buttons. A template
calls the method on each lookup, so every row made 7 virDomainGetInfo
RPCs (8 on a compute's instances page), and the memory column read the
domain XML once more.
- Instance.info reads virDomainGetInfo once per request; the lists and
instance_actions.html use it.
- The memory column of both index views comes from the same info
(current memory) instead of the domain XML. The value is the same:
checked against currentMemory for every running and shut-off VM of
the test hosts.
A row now makes one info call and one XML read (the title). The new
test counts the calls in the grouped and non-grouped index and on the
compute instances page; it fails on the previous code with 7 and 8
calls.
libvirt has no timeout for opening a connection. Measured in the test
setup: an unreachable host blocked qemu+tcp for 135 s and qemu+ssh for
133 s; a daemon that accepts and never answers blocks forever. Gunicorn
runs sync workers and the instance list connects to every compute, so
one unreachable compute held a worker for minutes per page load.
- A connection is opened in a helper thread, at most one per
connection at a time. The request waits LIBVIRT_CONNECT_TIMEOUT
(default 5 s). An open nobody waits for any more keeps running; until
it ends, requests get its error at once, and if it succeeds later the
connection is used. A failed open is not repeated for
LIBVIRT_RETRY_AFTER (default 30 s). A stuck host holds one thread and
does not keep other hosts from connecting.
- qemu+ssh runs ssh through vrtManager/libvirt-ssh (the URI's command
parameter), which adds ConnectTimeout=10 and BatchMode=yes, so an
abandoned ssh process ends and no prompt can hang. The user's ssh
configuration (aliases, ProxyJump, keys) still applies.
- The close callback acts only for the connection it was registered
for: a late callback of a replaced connection dropped its successor.
The reason is reported as text. A connection that closed before it
was published is an error.
- Compute.status is a bool (it returned the exception object, so the
"host is down" check in refresh_instance_database never fired); the
message is Compute.connection_error.
Unit tests cover the timeout, retry delay, late success, single open,
a stuck host next to a healthy one and the close callback. Live tests
(vrtManager/test_live_connection.py), for TCP and SSH computes of the
test setup: an unreachable address (no ssh left after 10 s), an endpoint
that never answers, a slow open completing later, a host that goes
silent on an established connection (keepalive ends calls within 45 s,
the next request reconnects), an ssh_config alias and a ProxyJump host.
The OpenStack-style cloud-init endpoints (index, meta_data.json,
user_data) are fetched by booting VMs, which do not log in. Since the
login-required middleware they redirected to the login page, so the
datasource has not worked. They are login_not_required now; they expose
a hostname and the public SSH keys of a VM's owners.
A VM is found by the reverse DNS name of the client address, and VM
names are not unique (only compute and UUID are). user_data merged the
owners of every VM with that name, so a user who named their VM like
another user's VM got their key into that VM at its next cloud-init
run. Keys are now given only when exactly one VM has the name.
The client address is the last X-Forwarded-For entry, which nginx in
the default setup appends; gunicorn listens on 127.0.0.1 only. The bare
except around the reverse lookup is narrowed to OSError.
SHOW_PROFILE_EDIT_PASSWORD was replaced in 2020 by the change_password
permission, settable per user or group. A post_migrate hook applied the
old setting to all users so existing installs kept their behavior, and
asked to remove the setting afterwards; but the settings template still
set it to True. Every migrate, so every upgrade, therefore granted the
permission to all users again and undid an admin's per-user or
per-group choice (with False it would have revoked it from everyone).
- Migration accounts.0008 applies the setting one last time, if a
settings.py still has it; the post_migrate hook is gone.
- New users get the permission when they are created (any path,
including LDAP); the admin user form ticks it and saves the admin's
choice afterwards.
- The template no longer sets SHOW_PROFILE_EDIT_PASSWORD; a check
(accounts.W001) asks to remove it from existing settings, and the
README upgrade notes say so.
The test that removes the permission and runs migrate fails on the
previous code. An upgrade from master with users was simulated: the
setting is applied once, and a permission removed afterwards stays
removed through the next migrate.
computes.views.get_compute_video_models had no URL and no caller, and
ComputeManager.get_video_models was used only by it. The video models
of the create page and the VM settings come from wvmConnect directly.
An optional dhcpd.conf next to the application supplies clone names and
their MAC addresses. get_dhcp_mac_address read it from BASE_DIR, but
guess_clone_name from a fixed /srv/webvirtcloud/dhcpd.conf, so on any
other install path the Guess button found no name while the MAC lookup
still worked. Both use utils.dhcpd_conf() (BASE_DIR) now.
The editor shows its Change button only for a shut-off VM, but
change_xml did not check the state: a page loaded while the VM was off
could still be saved after it was started, and the posted XML then
overwrote every change made to the definition in between. The view now
refuses unless the VM is shut off, with a message.
Creating each option on a RHEL qemu-kvm host showed:
- CD-ROM: the create page offered every disk bus, but the empty CD-ROM
of a new VM works only on ide, sata and scsi. fdc and virtio failed
("Invalid bus type", "does not support ejectable media"), usb failed
with a device named 'Nonea' because vol_dev_type() had no prefix for
usb, and after that libvirt refuses an empty usb disk. Only ide, sata
and scsi are offered now, and another value is refused before any
storage is allocated. vol_dev_type() names usb disks sdX, which also
fixes data volumes on the usb bus.
- USB controller: any input device added a bare <controller
type='usb'/>, which libvirt turns into piix3-uhci (USB 1.1) on q35
instead of its default qemu-xhci. The bare controller is gone; libvirt
adds the machine's default for USB inputs.
- MACs: with more networks than MAC addresses, create_instance indexed
past the list and the request failed with a 500. NICs without a given
MAC now get one from libvirt. MAC addresses are validated (format, not
more than networks) by the page and the API before anything is
created; they went into the XML unchecked.
Unit tests cover the refused values, the MAC rule and the usb prefix.
Live tests create a VM with every offered CD-ROM bus and start it,
check the USB controller with a USB input device and create a VM with
two networks and one MAC.
The form checked both with re.match("^[0-9a-f./:]+|^$"), which tests
only how a value starts: "2001:db8::10/64'/><evil/>" passed and went
into the interface XML, which is built with f-strings, while upper-case
hex was refused. An address without a prefix passed too and then failed
with a ValueError when the XML was built.
The address is parsed with ipaddress.IPv6Interface and stored as
address/prefix (/128 when none is given), the gateway with
IPv6Address. A static configuration needs an address, and the XML gets
a route only when a gateway is given instead of gateway=''. The HTML
patterns accept upper-case hex.
Eight of the ten form and XML checks fail on the previous code.
create_instance set secure='yes' (and with it SMM) only while switching
a non-q35 machine to q35. With q35 already chosen, a secure boot loader
was defined with secure='no' and without SMM. Secure boot is now
requested whenever the loader is a secure boot build; the switch to q35
is unchanged.
libvirt 11 on RHEL fills in secure='yes' and SMM from the firmware
descriptor of a known loader, so there the domain was already correct;
without such a descriptor (older libvirt, custom loader) it was not.
A unit test checks what WebVirtCloud requests (fails on the previous
code); a live test creates a q35 VM with the host's secure boot loader
and starts it.
create_instance passed the loader list where find_uefi_path_for_arch
expects the machine type. The domain capabilities were then read for
the default machine (pc) instead of the chosen one: on a RHEL host a q35
VM was offered /usr/share/OVMF/OVMF_CODE.secboot.fd, which q35 does not
list, instead of /usr/share/edk2/ovmf/OVMF_CODE.fd. The machine type is
passed now.
The page also read dom_caps["loader_enums"] into an unused variable;
on a host without loader support the key is missing and the page failed
with a KeyError. The line is removed.
A unit test covers the call and the host without loader support (it
fails on the previous code). A live test creates a q35 VM with the
offered UEFI firmware, checks that libvirt lists that loader for q35 and
starts the VM; it fails on the previous code.
attach_disk (new or existing volume, CD-ROM), add_guest_agent and
remove_guest_agent acted only when the status was running (1) or shut
off (5). On a paused VM they changed neither definition and the request
reported success. They now use the same rule as the NIC edits: the
running definition while the VM is active, paused included, and the
persistent one always. remove_guest_agent reads each definition from its
own XML, so a channel present only in the persistent one is removed too.
Unit tests check paused, running and stopped VMs (paused fails on the
previous code). The live effects test adds attach_disk and
add_guest_agent, and now also checks that edits meant for the running VM
reach it, which showed that an ejected CD-ROM keeps an empty <source>
in the live XML; NIC unplug is left to the guest tests, as these VMs
have no OS to acknowledge it.
New live guest tests compare what the guest agent reports (online vCPUs,
memory blocks, disk serials and buses, NIC MACs) with the persistent
definition, through the web views:
- at boot, with a data disk;
- after force off, CPU and memory resize and power on (2 vCPUs, 3 GiB);
- a hotplugged disk at once; a bus and serial change after powercycle;
- a NIC MAC change after powercycle, not before;
- a clone boots with its own MAC and disk copy;
- reverting a running snapshot removes a NIC added after it, in the
guest and in the definition, and leaves the VM running;
- a suspended guest does not answer and answers again after resume.
Every request in these tests must redirect (302), so a failing view is
not mistaken for a no-op. Only NICs with QEMU's 52:54: prefix are
compared, as a guest may create its own interfaces. All 13 guest tests
pass against a Rocky Linux 9 guest.
The live tests so far ran VMs without an OS, so everything that needs the
guest's cooperation was untested. With TEST_LIBVIRT_GUEST_IMAGE set to an
installed Linux image with qemu-guest-agent, instances/test_live_guest.py
boots wvc-test VMs from qcow2 overlays of it (the image is only read) and
checks through the web views:
- delete_vol on a running VM: the guest releases the disk, so it leaves
the live and persistent definitions and the volume is deleted;
- detach_vol removes the disk from the running guest and keeps the volume;
- add_network/delete_network: the guest sees the hotplugged NIC, and the
unplug completes in both definitions;
- set_vcpu brings a hotplugged vCPU online in the guest;
- osinfo returns the guest agent's OS, time zone and host name, and the
agent reports the VM's NIC by MAC;
- poweroff shuts the guest down through libvirt.
The image must be a BIOS-bootable qcow2 volume in a storage pool other
than wvc-test; the tests are skipped while a running VM uses it. They pass
against a Rocky Linux 9 guest on a RHEL qemu-kvm host; the guest had no
address on the NAT network (its network configuration is static), so only
the agent's NIC report is checked, not an address.
