The compute bootstrap found openEuler but had no install function for
it, so it stopped. Now it installs qemu, libvirt, python3-libguestfs
and cyrus-sasl (openEuler's names; it has no EPEL), applies the RHEL
family's qemu.conf, SASL and gstfsd steps, and runs the monolithic
libvirtd with its TCP socket: openEuler 24.03 ships modular daemons
only for qemu and the proxy.
Tested in an openEuler 24.03 container with /dev/kvm: the full live
suite over TCP (SASL) ran 2145 tests. What failed is libvirt 9.10's
(reverting a running VM's external snapshot, live clone needs 11.10),
the test host's (no pool named default), and its SeaBIOS, which
prints no banner on the serial port.
The removed openEuler branch only took 20.03, whose Python 3.7 cannot
run the panel. 24.03 LTS has Python 3.11: it installs like the RHEL
family, with the packages under their openEuler names (iproute holds
tc, cyrus-sasl the MD5 mechanism) and the RHEL family's main nginx
config. Other openEuler versions are refused with the supported one.
Tested on Ubuntu 22.04, 24.04 and 26.04, Debian 12 and 13, AlmaLinux 10
and openSUSE Leap 15.6. On Leap the panel did not start: the secret key
was written with python3 before the installer looked for Python 3.10+,
and Leap has only python3.11, so the key was empty. The installer still
said "Finished!" and exited 0, as it did for any failed step.
Now Python is found first. The package list update, each package, the
clone, the key, pip, migrate and collectstatic stop the installer with
a message when they fail (pipefail makes a logged step's status count).
Also:
- Ubuntu 26.04 is installed; Ubuntu 18/20 and Debian 9-11 are refused
up front (their Python is older than 3.10, so they failed later).
- The uos and openEuler branches are gone: untested, and their Python
is too old as well.
- Only `apt-get update`, no upgrade of the whole system.
- Debian no longer asks for smbios-utils (gone in Debian 13) and uuid,
which nothing uses; Debian and Ubuntu take python3-libvirt.
- RHEL reads the time zone like the others, and installs hostname for
ALLOWED_HOSTS.
- The unused wvcuser account is no longer created.
- The end of the run says where the admin password is.
It described Django 4.2 on Python 3.11, a distribution list the
installer does not support and no job worker. Now: the stack, the
processes and their units, what each app and vrtManager module holds,
how jobs claim, recover and share a host, and the API's routes; all
read from the code.
Behavior changes held one note per past fix; it keeps what an update
changes for operators and users. Migration is a section of bullets.
Docker cannot keep options that exist only in settings.py (LDAP,
OTP_ENABLED, LOGIN_TRUSTED_PROXIES): the README shows how to mount
one. The OpenLDAP examples use OpenLDAP names, the variables table
has ADMIN_USERNAME and ADMIN_PASSWORD, Contents lists every section,
and features name backups and the languages.
Removed: screenshots of the 2020 interface, the Gitpod badge and its
TODO-only config, and doc/readme.md, which held one word.
The daily backups stayed on the panel's host, so losing the host lost
them too. webvirt-offsite.timer now copies them at 03:45 with restic
over SFTP to another server, encrypted with a password only the
administrator and the host hold, and keeps 7 daily, 4 weekly and 6
monthly copies. It runs on the host as root, with Docker too, outside
the panel and its image. It fails when the newest backup is older
than 26 hours, so a stopped panel backup shows; an optional heartbeat
URL is called after each good copy.
The README has the setup, how to get a backup back from the copies,
and a fix found by the drill: moving to a new machine stopped when
the backup had no SSH keys (a panel without SSH computes).
The Backups section said only which files to put back on a new
machine. Now Restore and rollback has the steps, for Docker and for
installer and manual installations: restore the database, put back the
secret key, the worker's id and the SSH keys, and start. Users stay
signed in, the worker knows its old jobs and the computes accept the
panel again; the configuration outside the backup is brought over by
hand.
The only backups of the panel were the ones taken by hand before an
update. Now `manage.py backup` writes a new directory under backups/
with a consistent copy of the sqlite database (the backup API, while
the panel runs, checked with PRAGMA integrity_check), the secret key,
the worker's node_id, settings.py, the SSH keys of the panel's user
and, outside Docker, the commit. It keeps the newest 7 (--keep N) and
deletes older ones only after a good backup; a failed one leaves
nothing, and a second run while one runs fails. Files are mode 600,
directories 700: the database holds compute passwords and keys. With
PostgreSQL the database is left out.
It runs daily at 03:15: in Docker from the image's cron into a new
volume, webvirtcloud-backups; otherwise from webvirt-backup.timer,
which the installer and the manual steps install. A backup has the
layout of the update backups, so the restore steps take either.
A job's libvirt call to a host that is up but stuck waits as long as
the host does. Measured with the worker's loop: with two jobs stuck on
one host, the other host's jobs ran but their progress was never read
(both progress threads waited on the stuck host); with four stuck, a
new job of another host stayed queued, every slot taken; cancelling
them took all four control threads, so cancels, recovery and
housekeeping stopped for every host.
Now a host's jobs take at most half of the worker's slots (at least
one; a migration counts on both its hosts), and the next queued job of
another host starts instead. A host has at most one progress read and
one control task (a cancel or a recovery inspection) at a time, and
the control tasks of hosts leave one thread to housekeeping.
wvmInstance kept the guest agent's and the ARP table's addresses from
the first NIC it was asked about, and never read them again on that
object; it carried a lease lookup that was never filled.
The addresses are now one {MAC: (IPv4, IPv6)} read, kept like the
domain XML only inside a cached_reads block: the VM page, which reads
its NICs twice, asks the agent once (it may take its whole timeout),
and a read outside a page's block is a new one. The agent's addresses
still come before the ARP table's.
The worker wrote "A request ran past ... s" and then exited. A log write
can block, on a full pipe or on a handler lock a stuck request thread
holds; the worker would then not exit, and gunicorn's own heartbeat
timeout would end it only some ten minutes later, after the short claims
it holds had expired.
The line is written from another thread; the worker waits for it at most
a second, then exits.
A request counts on the host it connected to last. A quota check reads
every host of the user's VMs, and a VM's destruction the other hosts'
files, in the middle of a request that then goes on with the VM's own
host through the objects it kept: it was counted on the last host it
read while it used its own, so a host that does not answer could take
more requests than its limit, and the last host read was charged.
Both checks now give the request back to the host it counted on before.
vrtManager's errors (some 140: a disk that does not move, a network
address outside its subnet, a host device given to another VM, the
disks a migration cannot copy) were English f-strings, so no catalog
could translate them, on pages or in jobs.
They are marked texts with their values now; a variant that differs by
a word (internal or external, running or persistent, with a subnet or
without) is a whole message of its own, and a list of problems is a
list of messages. OperationError reads in the active language, so the
pages that show one translate it as they are; its args stay English, and
so do the logs and the audit log. Host device problems and the Migrate
tab's reasons a disk is not copied are messages too.
A job stored its error and its result's warning as English text, much
of it built with the values in it (volume paths, a disk, libvirt's
error), so no catalog could translate it, and the job pages and panel
showed English whatever the language.
A job now stores them as messages: our English text, marked for
translation, and its values; libvirt's own text is a message without
values and is shown as it is. The jobs API and pages put them in the
reader's language. OperationError keeps its text and values apart for
the same reason, so a refusal raised while a job is submitted is shown
translated too. Messages one after another still read as sentences.
Errors stored before are kept as one message each, shown as they were.
The job panel on VM and storage pages wrote its buttons and notes in
English (Cancel, Open, Refresh, Dismiss, Finishing..., Files not
removed, Worker unavailable, Cancel failed) and its state badge as the
state's code, and the jobs API gave the phase untranslated. The panel
now takes its texts and the state labels from the page's settings, and
the API gives the phase in the viewer's language, as it gives the
label; the job pages show that phase.
A job that cannot be cancelled and ran past its time limit had
"(over its time limit)" written into its phase, which then matched no
translation. The phase stays as the job sets it, and the pages say the
job is past its limit.
The VM page's and the Settings page's views marked their messages with
gettext_noop, so every message they gave (a refused resize, a saved
setting, a QoS change, a busy VM) showed in English whatever the
language, though the catalogs translate them. They translate in the
request's language now; their module-level texts lazily. The Settings
page's group titles stay English in the view, as the page links them
by their slugs and translates them itself.
Two values went into translated messages as English codes: a job's
state ("... is running on this resource") and a QoS direction ("Remove
the inbound limit ..."). They are named as the pages name them.
The Chinese catalog had not been extracted since 2023: of the panel's
1749 messages, 878 were missing and 455 carried old fuzzy guesses, and
some of those marked done said something else ("Delete API token" read
"Delete SSH key").
Every message is translated again, with one glossary for the panel's
terms, placeholders and markup checked against the English, and the
panel offers Simplified Chinese again. The CI check covers it too.
One message of the VM page (the console password's help) never reached
any catalog: its escaped quotes cut it short at extraction. It is
written with single quotes now, and translated in both catalogs.
The language choice on the Settings page listed every language Django
knows, 99 of them. Six catalogs (Czech under the wrong code, German,
Spanish, French, Russian, Ukrainian) translated none of the panel's
messages, so choosing one of them, like most of the others, showed
English. The Turkish catalog lacked the four newest messages.
The panel now offers English and Turkish (LANGUAGES). The empty
catalogs and the English one, which has nothing to translate, are
removed. Turkish translates every message again. CI checks that the
offered catalogs translate every message and that their compiled
files are up to date; the README says how to update them.
With threaded workers a request waiting on a stuck host holds one
thread, but each new request to that host (a VM list's poll every 5 s,
an F5) held another until the threads ran out.
In a process at most two web requests now use one host at a time,
counted by its name whatever the login. Another waits up to the connect
timeout (5 s) for one of them to end, and is then refused as the host
not answering: VM lists show the host as not reachable, pages show the
connection error. A request counts on the host it connected to last,
until it ends, so one that reads several hosts in turn holds none it
has left. Threads outside a request (the job worker's, the consoles')
are not counted: a job holds its host for as long as it runs.
A host that is up but does not answer (a hung NFS mount, for one) keeps
a libvirt call waiting: libvirt's keepalive does not end it, since the
daemon still answers keepalives. Each such request held a whole sync
worker, and once they held every worker the panel stopped answering,
pages without libvirt too; F5 added another.
gunicorn now runs threaded workers, 4 threads each, and a worker takes
no more connections than it has threads, so one whose threads are all
busy leaves new requests to the others. A threaded worker keeps
reporting in while a thread waits, so gunicorn's timeout no longer ends
the request: the worker now exits itself when a request runs past the
timeout (600 s), and gunicorn starts a new one. Short claims expire on
that bound.
The address search runs one at a time in a process, as it did with one
request per worker.
Two disks of a VM with the same file name in different directories
were both offered the same destination pool by default. The form was
sent, and the migration job refused it only then: both copies would
be one file.
The plan now gives each such disk, by default, a pool no earlier disk
of that name is copied into, and says under its choice that another
disk has the file name. A choice that still puts them in one pool is
refused by the job, as before.
A pool whose refresh failed, one busy with a volume copy for example,
was skipped without a word, so a hand-copied image stayed missing with
no reason given. The other pools are still refreshed; each failure is
shown with libvirt's error.
The API comment no longer says listing the ISO images refreshes pools.
The VM page refreshed every active storage pool to list the ISO images
for its CD-ROM drives. A pool holding large disk images took up to
seconds to refresh, most of the page's time, on every page load.
The page now lists the ISO images libvirt already knows. Images
uploaded through the panel are known at once. For an image copied to a
pool by hand, the Removable media section has a Refresh ISO list button
that refreshes the active pools and returns to the page.
The Content-Security-Policy now also sends script-src 'self': a script
injected into a page, inline or as an event handler, does not run. The
templates have none of their own any more, and a test keeps it so.
worker-src 'self' blob: lets the XML editor (Ace) and the API docs
(Redoc) start their workers, which they make from blob: URLs. The
Swagger UI page is drf-spectacular's split view, which serves its start
script as a file instead of inline.
The host pages lose their last inline scripts: the overview's charts
(host_graphs.js), the New pool, New network and New interface dialogs
(pool_dialog.js, network_dialog.js, interface_dialog.js), a network's
and a network filter's XML editors and dialogs (network_page.js,
nwfilter_page.js, nwfilters_page.js), the secrets page's labels
(secrets_page.js, its texts from the view as usage_texts) and a pool's
page with the ISO upload (storage_page.js). Their URLs and messages are
data attributes of the elements they belong to; the upload reads its
token and address from its form. No template has an inline script or
an inline event handler any more.
The VM lists lose their inline scripts and handlers: host_groups.js
collapses a host's VMs in the grouped list, new_vm_dialog.js reads the
new-VM dialog's CPU bars and capacity lines, and console_window.js
(loaded by every page) opens a console button's window, on the lists and
on the VM page alike. A list row's Start no longer calls the please-wait
dialog, which the list pages do not have (the call only raised an
error). The VM page's URL placeholders take the __name__ form the other
pages use.
The create wizard's two pages lose their inline scripts and handlers:
create_machine.js (machine types of the chosen architecture, Next, the
XML editor) and create_instance.js (volume and network pickers, disk
source, presets and flavors, row lists, a refused form restored). Their
URLs and messages are data attributes of #select_chipset, #next and
#create-vm-form; URL parts use __name__ placeholders, so the volume list
URL is now the routed one with its trailing slash. The XML form shows
the please-wait dialog through data-please-wait. The page script tests
read the static files.
The VM page's thirteen inline script blocks and its inline handlers move
to static files: instance_page.js (tabs, OS details, clone names and
MACs, Add disk dialog, boot order, host devices, migrate plan, console
window, switches that send their form), instance_stats.js (charts and
the Logs view) and instance_status.js (the reload when the VM's state
changes). Their values and messages come as data attributes of the
elements they belong to, page-wide ones on #vm-summary; a form marked
data-please-wait shows the please-wait dialog. Three blocks that did
nothing are dropped: popovers (the page has none), the video model
preselection (the server selects it) and the delete form's poll stop
(that form is on the destroy page). The HTML sink inventory now also
reads the app's own scripts.
A step towards a Content-Security-Policy without inline scripts. The
scripts every page or many pages share now load from static files: the
error focus of base.html, the please-wait dialog, the sign-in forms'
password toggle and busy button, the jobs pages' reload while a job
runs, and the start of the job panel and of the VM list's state poll,
whose values come as json_script data. Cancel links go back through a
data-history-back attribute instead of a javascript: URL. The IE8
html5shiv/respond.js block, X-UA-Compatible and shrink-to-fit are gone:
no supported browser reads them.
Both workflows ran on a push to any branch and on its pull request, so
every commit of a pull request ran the whole matrix twice, the two runs
sharing the runners: a pull request waited about ten minutes for its
checks. A push now runs them for master only; a pull request, also one
stacked on another, is tested by its pull_request run.
The buttons of the VM page's section bar and of the Settings bar under it
sat at the start of the row and left its end empty. Both bars now share
the whole row out among their buttons (Bootstrap's nav-fill), also when
they wrap on a narrow screen.
A template's page was a VM's: it opened on Power, with only a note there
that a template is not started, while its clone form waited in a section
of Settings, next to a console, access, snapshots and stats that a VM
that never runs has no use for. A template's page now carries a Template
badge and a Clone button, opens on the clone form, and leaves out Power,
Access, Snapshot, Stats, the console settings, host devices and the
guest agent and address lines; Resize, Settings and Destroy stay for
whoever may change it. A page that no section opens shows its first.
A VM made from a template came out at the template's size and shut off,
and was then resized and started by hand. The Clone tab now asks for the
clone's vCPUs and memory, filled in with the VM's, and whether to start
it. After the copy, before the clone is recorded, it is resized with the
Resize tab's own functions and started; a step that fails is a warning
of the job, and the clone stays. The quota counts the larger of the size
asked and the VM's, so a resize that fails leaves the clone at a size it
allowed.
When the job ends the page opens the clone's, not the cloned VM's again:
a template's page, shut off, read as a clone that had not started. An
outcome kept for a warning links to it.
A libvirtError made in a test takes the message of the thread's last real
libvirt error: in a live run, after another test's closed connection,
"no marker" read "client socket is closed" and the test failed. It now
checks that the move raised that very error, as the test after it does.
A VM list showed what it read when it was loaded: a VM stopped from
inside its guest or with virsh, or made or removed elsewhere, showed only
after a reload, unlike the VM page, which reloads when its VM's state
changes. The lists of all VMs and of one host's VMs now poll the states
of their VMs, at once and then every 5 s as the VM page does, and reload
when an answer differs from the one before. The states come from one
libvirt call per host; a user who does not see hosts gets only their own
VMs. The page in use (a dialog, a field, an unsaved XML edit, a job
outcome shown) holds the reload until it is free; that check is shared
with the job panel.
Since VM creation, clones, migrations and uploads run as jobs, the page
came back before they ended and the job panel then showed the outcome
with a Refresh button: a new VM, for one, showed only after a click. A
job of the page that succeeds without a warning now reloads the page,
which shows what it changed. The outcome box stays, with Refresh, for a
job that failed or warned, and when a reload would lose something: a
dialog open, a field in use, an XML edit not saved, or another job still
running, shown or awaited. A remembered job is taken when the page
renders, so the reload does not see it again.
Without a migration address libvirt sends a migrating VM to the
destination's own host name, which the source host often cannot resolve:
a live migration to a host added without one failed ("Unable to resolve
address ..."), also when the other host had one. The destination's
address on the Computes page, which the panel reaches it at, is now the
default; the migration address stays for another network. A host the
panel reaches on its loopback keeps libvirt's default.
A live migration whose source host could not resolve the destination's
name for the migration stream failed with "The host of this virtual
machine cannot be reached right now": libvirt reports that failure as a
connection error, and a job's error hides those behind that sentence so
as not to name hosts. Only superusers migrate, so the migration job now
fails with libvirt's own words, here "Unable to resolve address ...",
which points at the destination's Migration address.
The Migrate tab shows a running VM's live migration switch, on by
default. A VM shut down after the tab opened, as one is to move it shut
off, was then refused: "A shut-off VM cannot be migrated live". A
shut-off VM now moves offline whatever the request says about live, as
the tab tells for a shut-off VM; the tuning options are for a running VM
only, as before. An offline request for a running or paused VM is still
refused.
A shut-off VM with TPM state, as every Windows 11 VM, was refused for
migration: the state is a file of the source host, and an offline move
would start it there with a new TPM. Such a VM now migrates as a shut-off
VM with disks to copy does: its UEFI variables are copied, it is started
paused, its CPUs never run, migrates live, which carries the TPM state,
and is shut off on the destination. A shut-off VM with a saved state is
refused before any of that, for every way it moves: started paused, it
would resume and then lose the saved memory.
The clone tab says that a clone gets a new, empty TPM, and the destroy
page that the TPM state goes with the VM.
A VM made without a TPM, a Windows 10 to be upgraded to Windows 11, had
no way to get one but the XML editor. The Options tab has a TPM 2.0
switch, as the RNG one: it adds an emulated TPM of the host's model to
the definition for the next start, or removes it. Its state stays on the
host until the VM is undefined, so a TPM added again has it back. A TPM
the switch did not make (passthrough, 1.2, encrypted, with its own state
path) is left to the XML editor, and a host without swtpm says so.
A running VM got its new CD-ROM on SCSI, the one bus the panel could plug
in then, and Windows does not see a virtio-scsi drive before the driver
on that ISO is installed. Add CD-ROM is now a dialog that chooses the bus
and the ISO: a running VM takes USB, first, which every guest sees and
which hot-plugs, or SCSI; a shut-off one SATA (IDE on pc), SCSI or USB.
libvirt has no empty USB drive, so a USB CD-ROM takes its ISO when it is
added and is detached instead of ejected, also while the VM runs.
A USB CD-ROM is a CD drive on QEMU's usb-bot only. Where QEMU has none
(RHEL 9), libvirt took usb-storage, which shows the guest a hard disk
that Windows does not read; the drive now asks for usb-bot, and such a
host refuses it with libvirt's message.
nginx ends a proxied websocket that sends nothing for 60 seconds, and the
VNC console sends nothing while the VM's screen stays the same: an idle
console, an installer's progress screen, closed after a minute and had to
be connected again. novncd now pings the browser every 30 seconds. The SSH
tunnel of an SSH compute sends a keepalive as often, for a firewall on the
way that ends idle connections.
Windows 11 does not install without Secure Boot and a TPM 2.0, and the new
VM form offered neither: its firmware list was BIOS and one UEFI loader
guessed from file names, which on RHEL never matched the Secure Boot one.
The firmware choices are now BIOS, UEFI and UEFI Secure Boot, as the
host's libvirt reports them (its firmware features from libvirt 12.1,
before that its loaders), and libvirt picks the firmware itself: UEFI
without Secure Boot, or with Microsoft's keys enrolled. A TPM 2.0 switch,
shown when the host has swtpm, adds an emulated TPM. The create job
refuses a firmware or a TPM the host does not offer, and the API takes
both. The path-guessing helpers are gone.
A panel on another machine or in a container (the Docker setup, also on
the compute) reaches gstfsd at the compute's address, so most
installations need GSTFSD_BIND_HOST and GSTFSD_ALLOW_REMOTE; the step
named them only in its text. They now come with their own commands,
which replace any earlier lines, so a re-run or a half-set file ends with
the two.
When a VM stopped, its console's stream could end before the worker's
next check saw the VM shut off. The check then found no stream to drop and
said nothing, or, seeing the VM still running, opened the console again,
which libvirt refused, and the session ended. The worker now remembers
whether the VM ran at its last check, and a console that cannot be opened
again because the VM stopped meanwhile is a stop: the terminal says the VM
is not running and the console opens again when it starts. A new stream is
the console's before its callback is registered, which may end it at once.
The live serial test's VM now repeats its BIOS banner (it reboots right
after finding no boot disk), so a console attached late still gets one,
and the test waits for the superuser's console to show it before the VM
stops. A run of the whole live suite over the socket missed the banner of
the paused-then-resumed VM before.
The New pool form asks for the erase confirmation on a logical pool since
the storage pages were redone; the live test's pool request lacked it, so
the pool was never created and both LVM tests failed in setUp.
The Turkish catalog had not been extracted since 2024: of the panel's
1727 messages, 817 were missing and 489 carried old fuzzy guesses, so
most pages, dialogs and messages showed in English, and some fuzzy
guesses said something else.
The catalog is extracted again (obsolete entries dropped) and every
message is translated, the earlier terms kept (Sanal Makine, Sunucu,
Depolama havuzu, Birim, Anlık görüntü), placeholders and markup checked
against the English. The Settings page's names and descriptions come
from the database through migrations that did not mark them for
translation; they are marked now (gettext_noop, the stored values
unchanged), so they are in the catalog too.
Deleting or detaching a disk, deleting a network card, forcing a VM off,
reverting or deleting a snapshot, stopping or deleting a pool, a
network, an interface, a filter or a secret, and the other actions that
ask first opened the browser's yes/no box (confirm()), and the ISO
upload told its mistakes in the browser's alert().
One dialog in the base page now asks for all of them: its title and
confirming button carry the button's own words, its body the button's
question, the confirming button red for a destructive action. Confirmed,
the button is clicked again, so its form sends what it did before. A
button inside another dialog (the flavors) hides that one while asking
and shows it again after; Cancel gives the focus back, and the
confirming button acts once. The ISO upload's mistakes show in its
dialog's error box, announced.
