1
0
Fork 0
mirror of https://github.com/retspen/webvirtcloud synced 2026-11-08 11:16:47 +00:00
Commit graph

1626 commits

Author SHA1 Message Date
catborise
079e2ca5ca feat(instances): external snapshot delete runs as a job 2026-10-14 23:27:57 +03:00
catborise
c26419c757 feat(operations): recover interrupted jobs from libvirt evidence 2026-10-14 23:27:57 +03:00
catborise
a35284bbc9 feat(operations): the job worker 2026-10-14 23:27:57 +03:00
catborise
34582d23f3 feat(operations): worker concurrency and job retention settings 2026-10-14 23:27:57 +03:00
catborise
18146c30a1 feat(operations): submit, cancel, finish and resolve jobs 2026-10-14 23:27:57 +03:00
catborise
e5be5d4f71 feat(operations): kind contract, job context and worker lease 2026-10-14 23:27:57 +03:00
catborise
68812ddbdf refactor(computes): the lock helpers take claims instead of file locks 2026-10-14 23:27:57 +03:00
catborise
fca4a5d91c feat(operations): admit claims atomically; redact secrets 2026-10-14 23:27:57 +03:00
catborise
99b9aa2a3c feat(operations): job, claim and worker lease models 2026-10-14 23:27:57 +03:00
catborise
5f8f7a8643 fix(instances): a snapshot needs free space in the VM's storage pools
A disk-only snapshot was taken without any space check, though its
overlays grow as the guest writes and a later delete merges them down;
a full pool pauses the guest and can leave the snapshot undeletable.
Only a memory snapshot was checked, for the memory alone.

Before a snapshot, each storage pool with a disk of the VM must have
free a share of those disks' virtual size (10%, at least 1 GiB), and
the pool of the memory file the VM's memory on top. Both values are on
the Settings page (Snapshot Free Percent, Snapshot Free Min GiB; 0
turns a rule off, an invalid value keeps the default). The pools are
refreshed first, and a failed refresh or an unreadable volume refuses
the snapshot instead of skipping the check; a disk in no pool is not
checked. Nothing is reserved: later writes can still fill a pool.
2026-10-13 13:27:39 +03:00
catborise
f766890825 feat(instances): external snapshots can freeze the guest file systems
A disk-only snapshot of a running VM is crash-consistent: data the
guest has not written yet is missing from it. With a connected QEMU
guest agent, the snapshot form now offers to freeze the guest's file
systems while the disks are snapshotted (libvirt's QUIESCE flag); the
box is checked by default and can be cleared. It is offered only for a
running VM whose agent is connected, and not together with memory:
libvirt accepts quiesce only for disk-only snapshots, and a memory
snapshot keeps the whole guest state anyway. A request that does not
fit is refused before libvirt is called.
2026-10-13 10:55:24 +03:00
catborise
64ea12b0ce fix(instances): the revert confirmation says the VM restarts
Reverting a running or paused VM to a disk-only external snapshot
powers it off: libvirt starts it again from the snapshot's disks, as
after a power cut. The confirmation now says so for such a snapshot; a
snapshot with memory resumes the VM as it was saved.
2026-10-13 09:36:36 +03:00
catborise
5388743053 fix(vrtManager): refuse to migrate a VM with snapshots in every state
libvirt keeps snapshot metadata on the source host and a migration does
not carry it, even when both hosts share the disks. Only the shut-off
path refused such a VM itself; a running or paused one relied on
libvirt's own refusal. moveto now refuses it for every state, before
any check against the destination, with the same message.
2026-10-13 09:36:27 +03:00
catborise
097a91592c fix(instances): a snapshot file another VM uses is not called left behind
After a snapshot revert or delete, files libvirt could not remove are
named in a warning that says nothing uses them and asks to remove them
by hand. A file another VM on the same host uses (as a disk, a backing
file or a snapshot file) is no longer named: removing it would break
that VM. The warning now says no VM on this host uses the files.
2026-10-13 09:21:51 +03:00
catborise
c970f8e853 fix(storages): keep volumes a VM still needs
libvirt deletes any volume it is asked to. The storage page and the API
deleted a VM's disk, the base image of another volume or the file a
snapshot was taken on without a check, breaking the VM or its
snapshots.

wvmStorage.del_volume now refuses a volume a domain on the compute uses
as a disk or a backing file of one, a file of a domain's snapshots, or
the backing file of another volume. The check is the one destroy
already used, moved to wvmConnect and extended to snapshot files, so
destroy also keeps the files of another VM's snapshots. The storage
page shows the refusal; the API answers 400 with it, as for other
refusals. Deleting a disk from a VM refuses after the detach when the
VM's own snapshots need the volume, and says the volume was kept.

Network disks count by their source name, pool volume disks by their
volume's path. A volume whose metadata cannot be read stops the check
instead of counting as unused; only one removed meanwhile is skipped.
The check and the delete hold the compute lock, so no VM operation
attaches the volume in between; a busy compute is a message on the page
and 409 in the API.
2026-10-13 09:13:02 +03:00
catborise
08cad04468 fix(instances): deleting a VM's data takes its snapshot files too
Deleting a VM with its data deleted only the images its disks pointed
at. With external snapshots that is the newest overlay: the image the
snapshots were taken on, the older overlays and the memory files stayed
on disk, with nothing referring to them once the VM and its snapshot
metadata were gone.

The files the VM's snapshots were taken on, down each disk's backing
chain, the overlays of snapshots kept off the chain after a revert, and
the snapshots' memory files are now the VM's own too. A template the
VM grew from ends the chain and stays, as does an overlay of a disk
detached since. The backing references of the VM's own volumes no
longer count as another domain's use, and a file another VM uses keeps
the files below it.

A backing file that cannot be read (other than one in no pool) stops the
split before anything is deleted, instead of ending the chain there.
2026-10-13 09:03:05 +03:00
catborise
ff4ffab382 fix(instances): snapshot tab reads less and fails less
The snapshot tab listed the VM's snapshots three times per page: once
per kind and once more for all of them. The list of all is now made
from the two kinds, so the combined flag is no longer used anywhere.

The revert confirm named only the disks the persistent definition
would lose. A disk hot-plugged into the running VM alone is detached
too, so the live definition is compared with what ran when the
snapshot was taken.

A storage pool refresh error after libvirt had already made, reverted
or deleted the snapshot was shown as if the operation had failed. It
is now logged, and the left-behind check is skipped then, since a stale
pool still lists the removed files. A volume removed while that check
reads it is skipped too.

An error reading the block jobs no longer breaks the instance page;
create, revert and delete still query the jobs again and refuse.
2026-10-13 08:30:22 +03:00
catborise
3fd422db7e fix(instances): name snapshot files libvirt could not remove
When libvirt cannot remove a snapshot file it merged or discarded, it
only logs a warning and the delete or revert still succeeds. On NFS it
removes them as the qemu user, so an image directory the qemu user may
not write to left every merged overlay on disk without a word.

Revert and delete now compare the files in use before and after and
show a warning naming the ones still on disk. Only a snapshot's own
files (overlays, memory) and overlays on a snapshot point count, and a
disk the revert detached is never named, so a kept disk is not offered
for removal.
2026-10-12 21:41:20 +03:00
catborise
ff1508b3c2 feat(instances): external snapshots use libvirt's snapshot API
Create, revert and delete are one libvirt call each. The pflash-to-rom
loader rewrite, the hand-written block commit and the s1./s2. names are
gone, so UEFI VMs can take external snapshots, a revert keeps disks added
later, and deleting an older snapshot keeps the newer one.

Create refuses what libvirt could not revert or delete later: internal
snapshots on the VM, a saved state, a running snapshot job, disks that
are not files, CD-ROM media, a branch after a revert, and memory without
a running VM or room in the first disk's pool. A disk-only revert keeps
the VM running, paused or shut off; one with memory resumes it as saved.
A revert in the same second as the previous one, which libvirt fails
after stopping the VM, is retried once.
A snapshot the old code took of a UEFI VM saved the loader as rom; its
revert is refused (deleting it still works).

The snapshots tab lists internal and external snapshots separately, can
save memory too, shows running snapshot jobs, and the revert confirmation
names the disks it detaches and that NVRAM is not restored.
2026-10-12 15:25:44 +03:00
catborise
c55f52f025 test: each app keeps its tests in a tests package
131 test modules sat next to the code they test; instances alone had
51 of them beside views.py and utils.py. They move into a
tests/ package per app, and the JavaScript checks into
webvirtcloud/tests/js. Each non-empty tests.py becomes tests/test_views.py;
the empty generated stubs are removed.

Imports of the app's own modules become absolute, paths built from
__file__ go one level up, the live helper is instances.tests.livetest,
and coverage omits */tests/*. The suite discovers the same 960 tests as
before.
2026-10-12 15:17:30 +03:00
catborise
0906838193 fix(nginx): browsers revalidate static files
Static URLs carry no version, but nginx served them with expires max, so
after an update or a theme change browsers kept the old CSS and JS. They
now revalidate each load (304 when unchanged). CI checks the header, the
304 and that a changed file is sent again.
2026-10-12 08:52:41 +03:00
catborise
d4f04cbf1e build(deps): Bootstrap and Bootswatch 5.3.8
Bootstrap SCSS, the JS bundle and the Bootswatch themes go from 5.3.3 to
5.3.8 (bug fixes only), and wvc-main.min.css is recompiled from zephyr.
Bootswatch now gives table text the body color, and Lux drops card and
modal header borders.

Tests compile every theme with the pinned libsass and check that the
shipped CSS is zephyr from these sources. The update and rollback steps
put back the two files a theme change rewrites, or git refuses to pull
over them.
2026-10-12 08:44:31 +03:00
catborise
c23521bcf4 fix(appsettings): a theme change reaches the CSS nginx serves
nginx serves STATIC_ROOT (staticfiles/) since collectstatic moved there,
but the theme view wrote only static/css/wvc-main.min.css, so a new theme
showed up only after the next collectstatic. The view now also writes the
collected copy when collectstatic has run; the source copy stays so a
later collectstatic keeps the theme.
2026-10-12 08:36:43 +03:00
catborise
72681ab54e build(deps): the OTP e-mail draws its QR code with segno
django-qr-code was used for one thing: the QR code in the OTP e-mail,
as an inline SVG and a PNG. It brought pydantic, pydantic-core,
annotated-types and typing-inspection (about 10 MB) and, from 5.0 on,
Pillow. segno, which it used underneath, now draws both images with the
same options (error correction M, 4-module border, 1.8 mm SVG modules,
12 px PNG modules); the SVG, the PNG and the alt text are byte for byte
what django-qr-code 4.2 produced. account.html loaded the tag library
without using it.
2026-10-12 07:24:56 +03:00
catborise
99b9f35826 build(deps): drop linters that nothing runs
pycodestyle, pyflakes, pylint, yapf and black were only installed: CI,
pyproject.toml and the docs use ruff, which covers the pyflakes and
pycodestyle checks.
2026-10-12 07:24:56 +03:00
catborise
9030b910f5 build(deps): idna, requests and urllib3 above their advisories
django-bootstrap-icons (template tags; it downloads an icon missing from
static/icon_cache) and websockify depend on requests, unpinned. On a
host that already has python3-requests, the installer's venv kept the
distro's idna, requests and urllib3 (Rocky 10: idna 3.7, urllib3
1.26.19; Ubuntu 24.04: idna 3.6, requests 2.31.0, urllib3 2.0.7), all
with advisories, and the panel loads them with its templates. Checked
in fresh Rocky 10 and Ubuntu 24.04 containers with python3-requests
installed first: all three come from the venv.
2026-10-11 22:45:52 +03:00
catborise
3be3b4b2cc fix(install): openSUSE installs get python-ldap and pyasn1 without known flaws
The installer's venv sees the system site packages, and pip keeps a
system package that meets a requirement. On openSUSE Leap 15.6 the LDAP
login thus ran on the distro's python-ldap 3.4.4 (CVE-2025-61911, a
filter-escaping bypass, and CVE-2025-61912) and pyasn1 0.5.0
(CVE-2026-30922 and others), which django-auth-ldap loads through
ldap.controls. python-ldap>=3.4.5 and pyasn1>=0.6.4 make pip install
fixed copies in the venv. python-ldap builds there, so openldap2-devel
joins the openSUSE package lists (installer, manual and development
setup) and the update steps. Checked in a fresh Leap 15.6 container
with the installer's package list: 3.4.8 and 0.6.4 from the venv,
ldap.controls imports.
2026-10-11 22:45:52 +03:00
catborise
d706d58de4 build(deps): dev tools
coverage 7.16.2, pyflakes 4.0.3, pylint 4.1.3, black 26.10.1, ruff 0.17.0
(no new findings) and django-debug-toolbar 8.0.0; the template profiler
panel is pinned at 2.1.0. Toolbar 8 renders in a shadow DOM: the
profiler panel still opens and shows its data (checked in a browser).
Its script is requested by a relative URL and gets 404 with 6.3.0 too.
2026-10-11 21:09:32 +03:00
catborise
d3f3065e30 build(deps): django-otp 1.7.4
1.7.4 no longer preselects a device in the otp_device field. The login
form chooses the user's confirmed TOTP device itself and ignores that
field, so the login is unchanged (OTP login tests pass).
2026-10-11 21:09:32 +03:00
catborise
d86ab7d881 build(deps): django-bootstrap5 26.3
26.3 drops Django 4.2 and adds Django 6.1. The forms render the same
(compared on six pages). 26.4 needs Python 3.11; the app supports 3.10.
2026-10-11 21:09:32 +03:00
catborise
df4b1f4da6 build(deps): djangorestframework 3.18.3
3.18.2 hardens content negotiation against large Accept headers
(GHSA-33wh-fxxf-88vv) and refuses non-text charsets
(GHSA-3547-9m27-7rxg); the API reads both. 3.18 drops Django < 5.2 and
reports list serializer errors as a dict; the API's many=True
serializers only render output.
2026-10-11 21:09:32 +03:00
catborise
af0f67f712 build(deps): Django 5.2.18
Security release of 2026-10-06. Two of its fixes reach this app: a
quadratic HTTP header parsing DoS reachable without a login
(CVE-2026-84429) and long language codes through LocaleMiddleware
(CVE-2026-77050). The GeoDjango and model formset fixes do not apply.
2026-10-11 20:34:09 +03:00
catborise
87ffed66b9 fix(instances): attach a disk, NIC or guest agent to both definitions in one call
On an active VM a device was attached in two calls, running VM first.
When the persistent attach was refused, for example because a disk target
or an agent channel was already pending there, the device stayed in the
running VM only and was gone at the next start. A NIC added without a
MAC got a different generated MAC in each definition. One call with
AFFECT_LIVE|AFFECT_CONFIG lets QEMU check the persistent copy before the
hotplug and keeps one MAC. Detach and update still go per definition,
each from its own XML.
2026-10-11 20:12:24 +03:00
catborise
e924f3e7a0 fix(instances): a new disk on a running VM does not take a pending disk's name
The device name of a new disk or CD-ROM was chosen from the running VM
only. A name used only in the persistent definition, such as a disk
renamed by a pending bus change, could be chosen again: the disk was
attached to the running VM, the persistent attach was refused, and the
disk was gone after the next start. Names either definition uses are now
taken.
2026-10-11 17:58:19 +03:00
catborise
c1f6620c4b feat(instances): a running VM's settings tab says most settings apply at the next start
On an active VM most edits in the settings tab change the persistent
definition only. The tab now says so above its panes.
2026-10-11 17:47:44 +03:00
catborise
d0bd2ac32b fix(instances): a running VM's boot order, video and console forms show the pending settings
On an active VM the boot order editor, the video model select and the
console form showed the running VM: a pending boot order, video model or
console setting looked unset, and the boot order editor offered devices
of the running VM only. Choosing one of those cleared the pending order,
since the setter dropped the old order and skipped devices it could not
find. They now read the persistent definition (get_bootmenu,
get_bootorder, get_media_devices, get_video_model and the console
getters take config=), and set_bootorder refuses a device that is not in
it before it changes anything.
2026-10-11 14:56:51 +03:00
catborise
042530fbfd fix(instances): a running VM's options and CPU edits keep the pending changes
On an active VM the options and CPU forms write the persistent definition
but showed the running VM's title, description and vCPUs, so saving one
posted the live values back over a pending change. They now read the
persistent definition (get_vcpu, get_cur_vcpu, get_title and
get_description take config=).

Buttons in the settings and resize tabs that were disabled only by class
get the disabled attribute: Enter in a field no longer submits a form
the page shows as unavailable while the VM runs.
2026-10-11 14:09:23 +03:00
catborise
ea9424ee1f fix(instances): a running VM's QoS edit keeps the pending QoS changes
The QoS table showed the running VM's bandwidth while an edit changes the
definition for the next start. Each row posts average, peak and burst,
so an edit wrote the running values back over a pending change, and a
direction added while the VM ran had no row. The table shows the
persistent definition now. An edit also kept only the posted values and
dropped an inbound floor; it now updates them and keeps the rest.
2026-10-11 10:53:16 +03:00
catborise
8d2c9b1217 fix(instances): a running VM's disk edit keeps the other pending disk changes
On a running VM a disk edit changes the definition for the next start,
but the disk list and its edit form showed the running VM's disks. The
form posts every field, so an edit wrote the running values back over
earlier pending changes (cache, io, serial, ...), and a disk only in the
next definition could not be edited. The list now shows the persistent
disks. A volume is deleted only when both definitions use it at that
device, since the list may now show another source than the running VM.
2026-10-10 22:53:43 +03:00
catborise
0bb4337043 fix(interfaces): no create, start, stop or delete where libvirt cannot change interfaces
libvirt's udev interface backend (RHEL 9 and later, Debian 12) only reads
interfaces, so the create form and the start, stop and delete buttons
only failed there. The pages ask the host with an invalid definition,
which defines nothing: a backend without define refuses it as not
supported. There they show the interface state and a note instead, and
posts change nothing; the host's devices, needed only by the create
form, are not read. Any other answer keeps the pages as they were.
2026-10-10 21:38:22 +03:00
catborise
35fea424ee test: read the page scripts with lxml, not a tag regex
CodeQL flags the <script> regex (py/bad-tag-filter); the HTML parser finds the inline scripts as the new-VM dialog test does.
2026-10-10 21:04:38 +03:00
catborise
a57290cdf2 fix(instances): the new-VM dialog shows each host's current CPU load
The CPU bar showed the average since the host booted, read for every
compute on every load of the VM list. The dialog now reads each compute's
graph endpoint (load over one second) when it opens, two computes at a
time so the reads hold at most two workers, and ignores answers of an
earlier opening. A host that cannot be read shows a dash. The since-boot
reading had no other caller and is removed.
2026-10-10 20:54:19 +03:00
catborise
c1d2e9610d fix(instances): enabling vCPU hot plug keeps the current vCPU count
Enabling hot plug on a stopped VM wrote every vCPU up to the maximum as
enabled, so a VM with 2 of 4 vCPUs booted with 4. The current vCPUs stay
enabled now; libvirt adds the rest as disabled hotpluggable vCPUs.
2026-10-10 10:03:25 +03:00
catborise
251ddbbec1 fix(instances): a paused VM's memory is resized as a running VM's
Only a running VM counted as active, so a paused one (or one blocked,
PM-suspended or shutting down) had its maximum and current memory written
for the next boot and nothing applied live. Every active VM now has its
current memory changed live and in its definition and keeps its maximum;
a paused guest takes the new size once resumed. The quota check follows
the same rule. Checked on a real guest: paused, shrunk, resumed, then
grown back.
2026-10-10 09:52:56 +03:00
catborise
7b5c2cd057 perf(storages): ISO chunks to an SSH compute reuse the SSH connection
Each 32 MiB chunk is its own request and opened a new SSH connection and
SFTP session (~200 ms). A worker now keeps its connection to the host for
the next chunk; each request still opens its own SFTP session (~50 ms),
so an SFTP error never reaches the next one. A connection is not reused
after an error, when the host dropped it or once it is 10 minutes old; an
idle one is closed after a minute, and a forked worker leaves the parent's
to the parent. The SFTP start, which paramiko waits for without a deadline,
is ended after 10 s. Live: 256 MiB in 8 chunks 4.5-4.9 s -> 3.7 s.
2026-10-09 23:22:15 +03:00
catborise
ff2f16b226 style(tests): drop an unused import from the device edit tests
CI's Ruff job failed on it.
2026-10-09 22:25:13 +03:00
catborise
4ea002f5a3 perf(interfaces): list only the host's network devices
The interfaces page read the XML of every node device (183 on a test host)
to keep the network ones; libvirt now lists only those. Same names on both
test hosts, 756 -> 261 ms over TCP and 475 -> 173 ms over SSH.
2026-10-09 22:17:00 +03:00
catborise
f1e83455e4 refactor(instances): one way to reach a VM, its console and its devices
The .vv console file is built from the VM found by its UUID, with the
console page's getters, instead of looking the VM up by name five times;
a console without a listen address or password now gives 127.0.0.1 and
an empty password instead of "None". Removed methods nothing calls: the
by-name getters and power methods of wvmInstances, managed save, the vCPU
mode getter (it read the wrong attribute), the old NIC editor and its
bridge lookup. CD-ROM media and NIC hotplug use _definitions() like the
other device edits, the disk editor takes the bus prefix from
util.vol_dev_type, and an unreadable disk is logged, not printed.
2026-10-09 18:13:32 +03:00
catborise
0ea5f0ff18 fix(instances): a UEFI clone gets its own NVRAM without a new pool
Cloning a UEFI VM defined a persistent storage pool on the NVRAM's
directory when no pool held it, and dropped <nvram> from the clone,
which found its copy only because libvirt chose the same file name.
The copy is now made through the NVRAM directory's pool (a transient one
when there is none) or the pool that holds the file, with the original's
format, owner and mode, and the clone's definition names it. A failed
definition removes it with the copied disks. An NVRAM that is not a file
is refused before anything is copied.
2026-10-09 18:13:32 +03:00
catborise
11fe2409f9 fix(instances): device edits fit every definition and device type
A volume of an LVM or iSCSI pool can be attached: by its own path, with
the format its volume reports (raw when the pool knows none). Detaching
a disk takes each definition's own description of it, so a disk only in
the persistent definition goes too. Every NIC type can be a boot device;
bridge and direct NICs were dropped from the boot order. The video model
is a value, never parsed as XML. QoS changes clear the page's read cache.

The NIC model "default" leaves the choice to libvirt instead of writing
model "default", and rt18139 is rtl8139: a VM with either did not start
(checked on a RHEL 10 host). A migration corrects the default NIC type
setting.
2026-10-09 18:13:23 +03:00