A VM list row showed the VM's state, memory, vCPUs and title, read
through the VM's own connection object: a lookup by UUID, virDomainGetInfo
and the live XML, one after the other for every VM on the page.
The lists (Instances, grouped or not, and a compute's VMs) now read each
host once: the states in one getAllDomainStats call (the state group
only, as the lists' poll; no other group, so a VM busy in the QEMU
monitor holds nothing up) and the memory, vCPUs and title from each
VM's live XML, which the title already read. Checked on QEMU against
virDomainGetInfo for running, paused and shut-off VMs: the same values.
A VM gone from its host, or a host that cannot be read or fails its
listing, shows its VMs unknown as before. Only the grouped list, a
superuser's, prefetches every host's VMs and owners.
Every VM list load reconciles each compute with libvirt. Even when nothing
had changed, that took the compute's claims and a transaction with
SELECT FOR UPDATE (and the ownership advisory lock on PostgreSQL): about
11 queries a compute, 4 of them writes. On SQLite each load queued behind
every other write of the app.
The reconcile now first compares libvirt's VM list with the database,
without a claim or a transaction, and stops when they match and no
removed record has expired. Otherwise it runs as before, reading libvirt
again under its claims, so two reconciles still cannot commit in reverse
order. With two matching hosts a list load goes from 38 queries (8
writes) to 18 (none).
Nothing loads it: gunicorn (systemd, runit) and Django use
webvirtcloud.wsgi. It exec'd the shell script venv/bin/activate as
Python, which cannot run, and left that file open.
libvirt requires a stream that did not finish to be aborted before it is
freed. The NVRAM download of a shut-off move and upload_volume (NVRAM
upload, cloud-init seed) freed theirs after a failed send, receive or
finish without aborting, on the shared host connection. They abort now;
upload_volume opens its stream before it makes the volume, so the
cleanup has both.
The New password field sits under the "Set a new one" choice and shows
only while it is chosen, so its help no longer has to say so. The Console
heading above the Console type label goes, and the save button has the
size of the other tabs' main buttons.
The VM's XML editor takes most of the window's height and is framed like
a form field. The review dialog shows removed and added lines on a red and
green tint, and each part of the diff starts with its line number instead
of the unified diff's @@ header.
Unsafe migration and deleting the source files of copied disks sit below
a line, apart from how the memory moves, with their help in red. The disk
lines read "vda 10.0 GB" instead of a dot-joined fragment.
Random and Guess join the MAC field as one input group, the Clone tab's
secondary buttons are outlined like the other tabs', and the NIC and disk
lines read as plain text ("default pool, qcow2") instead of dot-joined
fragments.
Edit, Move, Detach, Unmount and a CD-ROM drive's Detach are outlined
secondary buttons and Delete an outlined danger one, as on the Network
and Snapshots tabs; Mount and Add disk stay the filled primary actions.
A NIC's row put its name and host device in one column, "Source",
"IPv4 unknown" and the link switch in another, and its model, MAC and
filter behind Device details; every NIC also had a "QoS limits · none"
fold and solid Edit, Add QoS and Delete buttons.
Now a row reads eth0, a cable, then the source: the cable is solid green
while the link is up and dashed grey while it is down. Under it are the
IPv4 addresses (or "No IPv4 address known") beside the link switch, and
a line of model, MAC, filter and whether bandwidth is limited. Device
details keep the type, the host device and IPv6; QoS limits show only
for a NIC that has one. The buttons are outlined. Dialogs, form names
and posts are unchanged.
The tab opened on "Snapshots · Active" above the form, and listed the
snapshots below it as rows of "name · description · external, current
· date" with a solid red Delete on each. Which snapshot the VM goes on
from was the word "current" among the others.
Now a line runs down the tab, newest first. On top is Now, the VM in
its state's color, with the form that takes a snapshot there. Below it
each snapshot is a point: its name and date, its description on a line
of its own, its kind under them. The one the VM goes on from is filled
and says so; the others are rings. Revert and Delete are small outlined
buttons. Forms, confirmations and the rules that disable them are
unchanged.
The tab listed seven settings at one weight, each a two-column card with
a heading and a paragraph beside its control. The controls disagreed:
three switches posted on change, while the video model, the crash action
and the guest agent (an Enabled/Disabled select) each had their own
Save, and "From the next start" was repeated under three of them.
Now it has three parts. VM details: title and description with their
Save, and the template switch below them. Devices the guest sees: video,
virtio RNG, TPM 2.0 and the guest agent as rows of name, control and a
one-line note, under one line saying when changes apply; the video model
posts on change and the guest agent is a switch, with its connection
shown beside it as the Power tab's dot. When the guest crashes:
Restart / Stop as two buttons that post on click. Form names, posts and
who sees what are unchanged.
The Turkish catalog now calls the guest agent "konuk ajanı" everywhere.
Each sub-tab (CPU, Memory, Disks) opened on a two-column card that put a
heading and a sentence of current values beside the forms, and the CPU
tab added two more rows: Live vCPUs as a row of "0 · Fixed / 1 · On"
buttons and vCPU hot plug as a select with its own Save.
Now each sub-tab opens on the VM's size. CPU: one cell per vCPU up to
the maximum, filled when on, with "2 on, up to 4" and the host's logical
CPUs; on a running VM with hot-pluggable vCPUs the cells are the
buttons that turn them on and off. Memory: a bar of the current memory
within the maximum, with the host's memory. Disks: a row per disk with
its format, size now and a new size. The forms below keep their names
and posts; the CPU fields are disabled while the VM runs, and hot plug
is a switch that posts on change. A memory or disk resize returns to
its own sub-tab.
The Resize tab offered vCPU counts only up to the host's logical CPUs.
A VM whose maximum was higher (4 on a 2-CPU host, say) found no option
for it, so the Maximum list showed 1, and a resize of the shut-off VM
lowered its maximum to that without a word.
get_max_cpus() now takes the VM's own maximum as a floor, and the
Instance's vcpu_range passes it. The host's CPU count shown on the page
and used by the clone form is read from the host alone, not from that
range.
The tab put the console, a remote viewer address and two root forms at
one weight under "Access · Active · VNC", named buttons after their
implementation (VDI, Get console.vv), and repeated why the root forms
were disabled under each of them.
Now it has two sections. Screen: Open console with what it does, and
the remote viewer on a row of its own (address, Open, Download .vv
file). Log in as root: one line says how a password or key reaches the
guest now, written to the disk of a shut-off VM or sent through a
running VM's guest agent, or what to do when neither works; the
password and key rows follow with aligned labels. Ids, field names and
posts are unchanged.
A checkbox marked a device as booting and also chose what Up and Down
moved, which moved every checked device at once: to move one device the
others had to be unchecked and checked again. The buttons sat apart
from the rows, and a VM booting by device type got a note that did not
say what to do.
Now the devices the VM tries are a numbered chain, each with its own
up, down and Remove, and the others are listed below it with Add; the
posted order is unchanged. Devices read by kind first (CD-ROM sda,
Disk vda) with their file or network below; a VM booting by device
type says so and how to choose an order. Autostart and the boot menu
sit below, lighter. Focus stays on the device that moved, and on a
narrow screen a device's buttons wrap below its name.
The Turkish catalog says "Ön yükleme" and "sanal makine" throughout,
also in the Power tab texts.
The tab put Power Off and Suspend, then Power Cycle and Force Off, in
joined button groups with a paragraph under each, so which sentence
told which button was left to the reader, and the two power cuts
looked like one control as large as the rest.
Now the tab shows the three states a VM moves between, Active,
Suspended and Off, the current one marked, and under each the action
that takes the VM there with one line on what it does. The power cuts
are small outlined buttons on a line of their own with one warning;
their confirmations stay. A state outside the three (shutting down,
crashed) is named above them. The test of the tab now finds the
outlined power-cut buttons, which its class match missed.
A quota refusal put the English name of the quota (cpu, memory, disk,
instance) into a translated sentence, so a Turkish page read "memory
kotası doldu". check_user_quota now returns which quota and its numbers
(QuotaExceeded), each quota has a whole sentence, and the refusals put
that sentence before what is refused; QUOTA_DEBUG adds the numbers as
before. The clone job stores the refusal as a message, so the Jobs page
shows it in its reader's language.
The connection errors of vrtManager (did not answer within, still
waited for, closed, failed with libvirt's text) were English f-strings;
they are messages now, each close reason a whole one. A connection
that died without a close callback raised an error reading "None"; it
now names the host. Logs keep the English text.
test_default_storage and test_default_storage_volumes opened a pool
named default, which a compute need not have (an openEuler host had
none, and the two tests failed there). They now use the wvc-test pool
the live suite creates and removes, and run only on a named test host.
On an AlmaLinux 10 host with SELinux enforcing, the installer finished
with exit 0 but the panel's four services never started: chcon labelled
the virtualenv httpd_sys_rw_content_t, which systemd may not execute
(status 203/EXEC, Permission denied), and the login page answered 502.
The README's manual steps did the same with httpd_sys_content_t. chcon
labels also went on a relabel, after which nginx got 403 for the
static files.
Now the installer writes file context rules, httpd_sys_content_t for
the panel and bin_t for venv/bin, restores the labels from them, and
does so whenever SELinux is enabled, not only when it enforces. The
manual steps add the bin_t rule and policycoreutils-python-utils (for
semanage).
The README said external snapshots need libvirt 9.9 or newer. The live
suite against an openEuler 24.03 LTS compute (libvirt 9.10) showed that
reverting a snapshot taken without memory of a running or paused VM is
refused there ("Invalid target domain state 'disk-snapshot'"); libvirt
11.10 on the test computes reverts it.
Opening /computes/<id>/storage/<pool>/ for a pool the host does not
have (or a host that does not answer) raised libvirt's error out of
the view: a 500. Now the pools list shows the reason, as the network
page does.
The compute bootstrap found openEuler but had no install function for
it, so it stopped. Now it installs qemu, libvirt, python3-libguestfs
and cyrus-sasl (openEuler's names; it has no EPEL), applies the RHEL
family's qemu.conf, SASL and gstfsd steps, and runs the monolithic
libvirtd with its TCP socket: openEuler 24.03 ships modular daemons
only for qemu and the proxy.
Tested in an openEuler 24.03 container with /dev/kvm: the full live
suite over TCP (SASL) ran 2145 tests. What failed is libvirt 9.10's
(reverting a running VM's external snapshot, live clone needs 11.10),
the test host's (no pool named default), and its SeaBIOS, which
prints no banner on the serial port.
The removed openEuler branch only took 20.03, whose Python 3.7 cannot
run the panel. 24.03 LTS has Python 3.11: it installs like the RHEL
family, with the packages under their openEuler names (iproute holds
tc, cyrus-sasl the MD5 mechanism) and the RHEL family's main nginx
config. Other openEuler versions are refused with the supported one.
Tested on Ubuntu 22.04, 24.04 and 26.04, Debian 12 and 13, AlmaLinux 10
and openSUSE Leap 15.6. On Leap the panel did not start: the secret key
was written with python3 before the installer looked for Python 3.10+,
and Leap has only python3.11, so the key was empty. The installer still
said "Finished!" and exited 0, as it did for any failed step.
Now Python is found first. The package list update, each package, the
clone, the key, pip, migrate and collectstatic stop the installer with
a message when they fail (pipefail makes a logged step's status count).
Also:
- Ubuntu 26.04 is installed; Ubuntu 18/20 and Debian 9-11 are refused
up front (their Python is older than 3.10, so they failed later).
- The uos and openEuler branches are gone: untested, and their Python
is too old as well.
- Only `apt-get update`, no upgrade of the whole system.
- Debian no longer asks for smbios-utils (gone in Debian 13) and uuid,
which nothing uses; Debian and Ubuntu take python3-libvirt.
- RHEL reads the time zone like the others, and installs hostname for
ALLOWED_HOSTS.
- The unused wvcuser account is no longer created.
- The end of the run says where the admin password is.
It described Django 4.2 on Python 3.11, a distribution list the
installer does not support and no job worker. Now: the stack, the
processes and their units, what each app and vrtManager module holds,
how jobs claim, recover and share a host, and the API's routes; all
read from the code.
Behavior changes held one note per past fix; it keeps what an update
changes for operators and users. Migration is a section of bullets.
Docker cannot keep options that exist only in settings.py (LDAP,
OTP_ENABLED, LOGIN_TRUSTED_PROXIES): the README shows how to mount
one. The OpenLDAP examples use OpenLDAP names, the variables table
has ADMIN_USERNAME and ADMIN_PASSWORD, Contents lists every section,
and features name backups and the languages.
Removed: screenshots of the 2020 interface, the Gitpod badge and its
TODO-only config, and doc/readme.md, which held one word.
The daily backups stayed on the panel's host, so losing the host lost
them too. webvirt-offsite.timer now copies them at 03:45 with restic
over SFTP to another server, encrypted with a password only the
administrator and the host hold, and keeps 7 daily, 4 weekly and 6
monthly copies. It runs on the host as root, with Docker too, outside
the panel and its image. It fails when the newest backup is older
than 26 hours, so a stopped panel backup shows; an optional heartbeat
URL is called after each good copy.
The README has the setup, how to get a backup back from the copies,
and a fix found by the drill: moving to a new machine stopped when
the backup had no SSH keys (a panel without SSH computes).
The Backups section said only which files to put back on a new
machine. Now Restore and rollback has the steps, for Docker and for
installer and manual installations: restore the database, put back the
secret key, the worker's id and the SSH keys, and start. Users stay
signed in, the worker knows its old jobs and the computes accept the
panel again; the configuration outside the backup is brought over by
hand.
The only backups of the panel were the ones taken by hand before an
update. Now `manage.py backup` writes a new directory under backups/
with a consistent copy of the sqlite database (the backup API, while
the panel runs, checked with PRAGMA integrity_check), the secret key,
the worker's node_id, settings.py, the SSH keys of the panel's user
and, outside Docker, the commit. It keeps the newest 7 (--keep N) and
deletes older ones only after a good backup; a failed one leaves
nothing, and a second run while one runs fails. Files are mode 600,
directories 700: the database holds compute passwords and keys. With
PostgreSQL the database is left out.
It runs daily at 03:15: in Docker from the image's cron into a new
volume, webvirtcloud-backups; otherwise from webvirt-backup.timer,
which the installer and the manual steps install. A backup has the
layout of the update backups, so the restore steps take either.
A job's libvirt call to a host that is up but stuck waits as long as
the host does. Measured with the worker's loop: with two jobs stuck on
one host, the other host's jobs ran but their progress was never read
(both progress threads waited on the stuck host); with four stuck, a
new job of another host stayed queued, every slot taken; cancelling
them took all four control threads, so cancels, recovery and
housekeeping stopped for every host.
Now a host's jobs take at most half of the worker's slots (at least
one; a migration counts on both its hosts), and the next queued job of
another host starts instead. A host has at most one progress read and
one control task (a cancel or a recovery inspection) at a time, and
the control tasks of hosts leave one thread to housekeeping.
wvmInstance kept the guest agent's and the ARP table's addresses from
the first NIC it was asked about, and never read them again on that
object; it carried a lease lookup that was never filled.
The addresses are now one {MAC: (IPv4, IPv6)} read, kept like the
domain XML only inside a cached_reads block: the VM page, which reads
its NICs twice, asks the agent once (it may take its whole timeout),
and a read outside a page's block is a new one. The agent's addresses
still come before the ARP table's.
The worker wrote "A request ran past ... s" and then exited. A log write
can block, on a full pipe or on a handler lock a stuck request thread
holds; the worker would then not exit, and gunicorn's own heartbeat
timeout would end it only some ten minutes later, after the short claims
it holds had expired.
The line is written from another thread; the worker waits for it at most
a second, then exits.
A request counts on the host it connected to last. A quota check reads
every host of the user's VMs, and a VM's destruction the other hosts'
files, in the middle of a request that then goes on with the VM's own
host through the objects it kept: it was counted on the last host it
read while it used its own, so a host that does not answer could take
more requests than its limit, and the last host read was charged.
Both checks now give the request back to the host it counted on before.
vrtManager's errors (some 140: a disk that does not move, a network
address outside its subnet, a host device given to another VM, the
disks a migration cannot copy) were English f-strings, so no catalog
could translate them, on pages or in jobs.
They are marked texts with their values now; a variant that differs by
a word (internal or external, running or persistent, with a subnet or
without) is a whole message of its own, and a list of problems is a
list of messages. OperationError reads in the active language, so the
pages that show one translate it as they are; its args stay English, and
so do the logs and the audit log. Host device problems and the Migrate
tab's reasons a disk is not copied are messages too.
A job stored its error and its result's warning as English text, much
of it built with the values in it (volume paths, a disk, libvirt's
error), so no catalog could translate it, and the job pages and panel
showed English whatever the language.
A job now stores them as messages: our English text, marked for
translation, and its values; libvirt's own text is a message without
values and is shown as it is. The jobs API and pages put them in the
reader's language. OperationError keeps its text and values apart for
the same reason, so a refusal raised while a job is submitted is shown
translated too. Messages one after another still read as sentences.
Errors stored before are kept as one message each, shown as they were.
The job panel on VM and storage pages wrote its buttons and notes in
English (Cancel, Open, Refresh, Dismiss, Finishing..., Files not
removed, Worker unavailable, Cancel failed) and its state badge as the
state's code, and the jobs API gave the phase untranslated. The panel
now takes its texts and the state labels from the page's settings, and
the API gives the phase in the viewer's language, as it gives the
label; the job pages show that phase.
A job that cannot be cancelled and ran past its time limit had
"(over its time limit)" written into its phase, which then matched no
translation. The phase stays as the job sets it, and the pages say the
job is past its limit.
The VM page's and the Settings page's views marked their messages with
gettext_noop, so every message they gave (a refused resize, a saved
setting, a QoS change, a busy VM) showed in English whatever the
language, though the catalogs translate them. They translate in the
request's language now; their module-level texts lazily. The Settings
page's group titles stay English in the view, as the page links them
by their slugs and translates them itself.
Two values went into translated messages as English codes: a job's
state ("... is running on this resource") and a QoS direction ("Remove
the inbound limit ..."). They are named as the pages name them.
The Chinese catalog had not been extracted since 2023: of the panel's
1749 messages, 878 were missing and 455 carried old fuzzy guesses, and
some of those marked done said something else ("Delete API token" read
"Delete SSH key").
Every message is translated again, with one glossary for the panel's
terms, placeholders and markup checked against the English, and the
panel offers Simplified Chinese again. The CI check covers it too.
One message of the VM page (the console password's help) never reached
any catalog: its escaped quotes cut it short at extraction. It is
written with single quotes now, and translated in both catalogs.
The language choice on the Settings page listed every language Django
knows, 99 of them. Six catalogs (Czech under the wrong code, German,
Spanish, French, Russian, Ukrainian) translated none of the panel's
messages, so choosing one of them, like most of the others, showed
English. The Turkish catalog lacked the four newest messages.
The panel now offers English and Turkish (LANGUAGES). The empty
catalogs and the English one, which has nothing to translate, are
removed. Turkish translates every message again. CI checks that the
offered catalogs translate every message and that their compiled
files are up to date; the README says how to update them.
With threaded workers a request waiting on a stuck host holds one
thread, but each new request to that host (a VM list's poll every 5 s,
an F5) held another until the threads ran out.
In a process at most two web requests now use one host at a time,
counted by its name whatever the login. Another waits up to the connect
timeout (5 s) for one of them to end, and is then refused as the host
not answering: VM lists show the host as not reachable, pages show the
connection error. A request counts on the host it connected to last,
until it ends, so one that reads several hosts in turn holds none it
has left. Threads outside a request (the job worker's, the consoles')
are not counted: a job holds its host for as long as it runs.
A host that is up but does not answer (a hung NFS mount, for one) keeps
a libvirt call waiting: libvirt's keepalive does not end it, since the
daemon still answers keepalives. Each such request held a whole sync
worker, and once they held every worker the panel stopped answering,
pages without libvirt too; F5 added another.
gunicorn now runs threaded workers, 4 threads each, and a worker takes
no more connections than it has threads, so one whose threads are all
busy leaves new requests to the others. A threaded worker keeps
reporting in while a thread waits, so gunicorn's timeout no longer ends
the request: the worker now exits itself when a request runs past the
timeout (600 s), and gunicorn starts a new one. Short claims expire on
that bound.
The address search runs one at a time in a process, as it did with one
request per worker.
Two disks of a VM with the same file name in different directories
were both offered the same destination pool by default. The form was
sent, and the migration job refused it only then: both copies would
be one file.
The plan now gives each such disk, by default, a pool no earlier disk
of that name is copied into, and says under its choice that another
disk has the file name. A choice that still puts them in one pool is
refused by the job, as before.
A pool whose refresh failed, one busy with a volume copy for example,
was skipped without a word, so a hand-copied image stayed missing with
no reason given. The other pools are still refreshed; each failure is
shown with libvirt's error.
The API comment no longer says listing the ISO images refreshes pools.
The VM page refreshed every active storage pool to list the ISO images
for its CD-ROM drives. A pool holding large disk images took up to
seconds to refresh, most of the page's time, on every page load.
The page now lists the ISO images libvirt already knows. Images
uploaded through the panel are known at once. For an image copied to a
pool by hand, the Removable media section has a Refresh ISO list button
that refreshes the active pools and returns to the page.
The Content-Security-Policy now also sends script-src 'self': a script
injected into a page, inline or as an event handler, does not run. The
templates have none of their own any more, and a test keeps it so.
worker-src 'self' blob: lets the XML editor (Ace) and the API docs
(Redoc) start their workers, which they make from blob: URLs. The
Swagger UI page is drf-spectacular's split view, which serves its start
script as a file instead of inline.
The host pages lose their last inline scripts: the overview's charts
(host_graphs.js), the New pool, New network and New interface dialogs
(pool_dialog.js, network_dialog.js, interface_dialog.js), a network's
and a network filter's XML editors and dialogs (network_page.js,
nwfilter_page.js, nwfilters_page.js), the secrets page's labels
(secrets_page.js, its texts from the view as usage_texts) and a pool's
page with the ISO upload (storage_page.js). Their URLs and messages are
data attributes of the elements they belong to; the upload reads its
token and address from its form. No template has an inline script or
an inline event handler any more.
The VM lists lose their inline scripts and handlers: host_groups.js
collapses a host's VMs in the grouped list, new_vm_dialog.js reads the
new-VM dialog's CPU bars and capacity lines, and console_window.js
(loaded by every page) opens a console button's window, on the lists and
on the VM page alike. A list row's Start no longer calls the please-wait
dialog, which the list pages do not have (the call only raised an
error). The VM page's URL placeholders take the __name__ form the other
pages use.
