- libvirt errors all answered 400. A VM, pool or volume the host does not
have now answers 404, a volume in use 409 (vrtManager.util.InUse) and a
host that cannot be reached 503, which a client may retry.
- A template's power on or cycle and a stopped pool answered
{"status": ...} with 409; they answer {"detail": ...}, as every other
refusal. The schema documents these 409s (ErrorSerializer).
- The volume list and a volume of a stopped pool answered 200 with an
empty list or a made-up volume; they answer 409.
- An API URL that names nothing answered the HTML error page; it answers
a JSON 404 after the API's authentication.
- A session that must change its password was redirected to the HTML
form; an API request of it gets the JSON 403 a token gets.
English-only texts users read, now in their language:
- the serial console's four refusals, which socketiod sends in the
language LocaleMiddleware would choose (cookie, then browser) and logs
in English; the texts live in console/refusals.py, as makemessages
reads only .py files;
- a submission key reused for another request, a malformed MAC address,
the moved-VM refusal of a script's request, the API's "Templates
cannot be started" (now the page's sentence), "Pool is not active"
and the ISO upload's checks;
- "Problem occurred with host" was a fragment with the host's name and
error after it: one sentence with both.
- A failed cancel hook logged "worker task cancel failed": it now names
the job, its kind, resource and user, with the traceback for a fault
of the panel.
- novncd logged "Console connection failed" without the VM or the user.
- django-axes wrote four INFO lines a login ("Reset 0 access
attempts"...); logins are on the Logs page, so it logs warnings only.
- The new VM form's machine types and disk buses answered 200 with an
empty object when libvirt failed, and nothing was logged: the list
stayed empty, the form could not go on and said nothing. They answer
400 with the error, as the pool's volume list does, the warning is
logged with the compute, and the form's lists show an error they get
with the page's messages.
A job that failed on an exception that is not libvirt's or a refusal
showed its raw text to users (a SQL error such as "UNIQUE constraint
failed: instances_instance.uuid", a KeyError's key), and the worker
logged only that text. Users now read "The panel failed unexpectedly;
its log has the details." and the worker logs the exception with its
traceback.
A live clone whose recovery found the clone made but could not thaw the
source's file systems kept that in the error of a succeeded job, which
the job panel does not show. It is a warning of the clone's result, as
when the job itself made the clone.
Since the VM create became a job (6f0c7300) every new VM was defined with
no VNC password, and a clone copied its source's password (or its lack
of one) exactly: every clone of a template shared it. Computes listen
for VNC on all their addresses (dev/libvirt-bootstrap.sh), so anyone who
could reach a compute's VNC ports opened any of these consoles directly,
without the panel; the README's firewall rule was the only guard.
The create job asks for create.py's random 8-character password again,
and a clone's VNC graphics get a new random one. The console page sends
the password to a user with passwordless_console, which the user create
form now ticks by default next to change_password; a user without it is
asked for the password, as the permission intends. Existing VMs keep
their settings.
Live create, clone and VM page tests pass on a TCP compute.
Since deleting a VM became a job (#431), the delete page queues it and
returns to the VM page; the live VM page tests still expected the VM
gone at once and a redirect to the list, so they failed, and the VMs
they left behind failed later tests. They now run the queued job and
check the VM is gone.
- console settings were split over four files; who sees the form, the
listen address, and the settings themselves are test_console_settings,
and who may open a console is test_console_access;
- the three template files are test_templates;
- the quota lock test joins test_quota;
- accounts' test_security_fixes held two topics: the SSH key tests are
test_ssh_keys, the owner-row delete redirect is in test_views;
- instances' test_security_mutations, who may change a VM, is
test_permissions.
The suite runs the same 2097 tests.
Tests that build vrtManager objects and touch no view, form or model of
an app lived in that app; vrtManager/tests already holds such tests, so
the convention was split. They move to vrtManager/tests: the firmware,
CPU mode and PCIe port definitions of a new VM (test_create), USB and PCI
host devices, the console listener, host CPU reads, interface reads,
deletes and XML, network filters, PCI passthrough rules, node devices,
network changes, addresses and leases, pool XML, secrets, and the live
tests that use vrtManager only (disk resize, ISO, snapshots, XML, live
clone). livetest.py, the live-test helper six apps import, and the
network test driver move with them.
Others go to their app: the quota with a host down to instances, the
401/403 Logs rows (webvirtcloud's middleware) to webvirtcloud, the VM
page's tab script test to instances, novncd's own tests out of the
console views tests, the compute lock and input validation tests out of
the computes views tests, the interface views out of the forms tests.
instances/tests/test_views.py, entirely live, is test_live_views.py.
Test modules imported fixtures from each other; shared ones are now in
vrtManager/tests/fakes.py (migration fakes, node devices, IOMMU groups,
leases, a NAT network, a Ceph secret), interfaces/tests/fakes.py,
console/tests/daemons.py (novncd loaded once), livetest.wait_for and
instances/tests/create_page.BODY. Every live test checks
livetest.enabled(). The suite runs the same 2097 tests.
Ten classes ran on TestCase or TransactionTestCase although they read
files, templates or scripts only. SimpleTestCase skips the transaction
and refuses a database query, so a test that starts to use the database
says so. LoggingConfigTestCase also removes the temporary directory it
makes.
- create_admin had a branch for `"test" in sys.argv` that gave the admin
the password "admin"; 17 places signed in with it, and failed silently
with ADMIN_PASSWORD set in the environment. The tests now make their
own superuser and force_login it, and the shipped code has no test
branch: the test runner gives migrate a password (also when
ADMIN_PASSWORD is set but empty), so no data/admin_password is written
into the checkout.
- AccountsTestCase builds its rows in setUpTestData instead of
setUpClass and deleting them by hand.
- accounts tests set OTP_ENABLED with override_settings instead of
assigning settings (one assignment was never undone, so later tests in
the process ran with OTP on), and undo their app_settings change. The
OTP login class reloads the URLs again only after the setting is off.
- computes/tests/test_views.py reached the machine's own libvirt
(localhost over the socket) when TEST_LIBVIRT_HOST was unset; its live
tests now run only with TEST_LIBVIRT_HOST, like the other live tests,
as the README says.
Networks, Interfaces, Storages, Storage, Volume, MigrateInstance and
CreateInstance were unmanaged models with no table, there only so that
ModelSerializers and the new VM ModelForm could take their fields. They
made content types and migrations, and their DO_NOTHING foreign keys
added reverse accessors that meant nothing (Compute.host,
Compute.target_host, Instance.source_host).
The serializers are plain Serializers and NewVMForm a Form, field for
field as the models gave them (types, lengths, required, defaults, the
minimum of 1 for vCPUs, memory and disk size; the form's labels are
still derived from the field names). Compared field by field before and
after, the API schema changes only where the models were wrong:
- no int64 bounds on the listing fields;
- the interface fields lose the titles that named the wrong column
(type was "Status", state "Device", mac "Forward");
- a pool's volumes is a list of objects (it was described as an
integer) and a volume's allocation an integer (it was untyped).
One migration per app deletes the models; none touches a table.
instances/forms.py and instances/models.py imported values from the
webvirtcloud.settings module itself, which bypasses DJANGO_SETTINGS_MODULE
(settings-dev, test overrides) and runs the module a second time. They
now read django.conf.settings. webvirtcloud/urls.py also loses its
fallback that set REST_FRAMEWORK's schema class at import for settings.py
copies without it: the template sets it, and an older settings.py is
rebuilt from the template when updating.
The root password hash for a shut-off VM used crypt_r where installed
(Python 3.13) and fell back to the stdlib crypt, which Python 3.11 and
3.12 deprecate: `python -W error::DeprecationWarning manage.py check`
failed there. crypt-r (the same module, it requires Python 3.11) is now
installed from 3.11, and the module is chosen by the Python version;
3.10's crypt is not deprecated.
- noVNC's own web client (app/ except the bell sounds, vnc.html,
vnc_lite.html, defaults.json, mandatory.json): the console imports
only noVNC's core, and novncd serves no web pages. AUTHORS, LICENSE.txt
and package.json (the version) stay.
- 24 cached icons no {% bs_icon %} asks for; the icon cache test now
also fails on an icon nothing uses.
- Ace's ext-textarea.js, which nothing loads.
- instances/fixtures/initial_data.json, for an app ("create") that no
longer exists; migration 0006 seeds the flavors.
- webvirtcloud/asgi.py: gunicorn runs the WSGI application only.
- instances/templatetags/tags_active.py: a copy of common_tags'
class_active that no template loads.
- .github/linters/.hadolint.yaml, from super-linter, which no workflow
runs anymore.
Each was checked with git grep across every tracked file (templates, JS,
shell, unit files and the extensionless console daemons) and for
dynamic use (URL names, template names, registries):
- vrtManager: host_is_up, is_kvm_supported, get_emulators,
get_file_extensions, both get_network_forward, get_snapshots,
get_storages_images, get_os_type, get_host_arch, the get_pretty_* trio
and, in util, is_kvm_available, randomMAC, get_max_vcpus, compareMAC,
pretty_mem, pretty_bytes and validate_uuid; with them the port
constants only host_is_up read. Some were reached only by their tests,
which go too.
- The flavor edit page: nothing linked to it; flavors are created and
deleted on the Flavors page and edited through the API.
- computes.validate_name, UserForm.identity_has_errors, the has_perm
template tag, the ALLOW_EMPTY_PASSWORD setting nothing reads,
hidePleaseWaitDialog, default_app_config (ignored since Django 4.1)
and the Python < 3.6 lines of secret_generator.py.
The delete ran in the request: undefine, then the VM's record, then its
disk files. A request that died in between (a gunicorn worker past its
timeout, OOM, a restart), or a database write that failed after the
undefine, left a record that later showed as removed outside the panel,
and the disk files the user asked to delete were never deleted.
Deleting a VM is now an instance.delete job. It holds the compute
exclusively, as the request did, so no other VM attaches a file it is
about to delete. The job chooses the files before it stops the VM (a host
that cannot be checked still stops nothing) and records them first. It
then stops and undefines the VM, removes its record, and deletes the
files, each with its own Logs row. Recovery decides from libvirt: a VM
still on its host means the delete failed; a VM gone means libvirt did
it, so recovery removes the record and deletes the recorded files that
are still there.
The page queues the job and returns to the VM page, where the job panel
follows it and opens the VM list when it ends; disks that were kept are
the result's warning. The API's delete answers 202 with the job and
needs a submission_key. backing_path moves to wvmConnect, as recovery
has no domain to ask.
The worker sends cancels only to jobs its own threads run. After a
worker restart, a migration or live clone that libvirt still ran was
watched by recovery; a cancel then moved the job to cancelling, the page
said so, but nothing told libvirt, and the job ran to its end. Recovery
now sends the cancel once while libvirt still runs the job, and a
migration it aborted ends cancelled rather than failed.
migrate_instance sets autostart on the destination (it is per host and
the migration does not carry it) and deletes the NVRAM file libvirt
keeps on the source. A worker that stopped after libvirt moved the VM
skipped both: recovery only recorded the move, so an autostart VM no
longer started after the destination rebooted, and a later VM of the
same name on the source started with these UEFI variables. The job now
records both before the move, and recovery does them after a proven
move; a step that fails is a warning of the job's result.
A shut-off VM with a TPM or disks to copy migrates started paused
(migrate_running), and the job shuts it off afterwards on whichever host
has it. A worker that stopped in between left it running paused, on the
source when the migration did not happen or on the destination when it
did; recovery reported the outcome but left the VM running. The job now
records whether the VM was shut off, and recovery destroys it again
where it ended up.
After a successful copy, recovery defined the clone; a define that
failed raised, and recovery retried it with the same result. One case
is certain: a worker killed after define_clone copied the UEFI
variables but before it defined the VM leaves <clone>_VARS.fd, and every
retry is refused because the volume exists. The job then waited for
recovery with its claims on the source VM and the clone's name. A
libvirt error now ends it uncertain, naming the error and the copied
volumes; an unreachable host still waits.
A CD-ROM form sent twice (a double click, a resent request) added a
second drive with the same ISO. The form is refused when the ISO is in a
CD-ROM of the VM's running or persistent definition; an empty drive may
still be added more than once. tr and zh too.
The workbook kept every cell as an object: a 100 000-row Logs download
took some 400 MB in a web worker. openpyxl's write-only mode writes the
rows as they come (85 MB for the same file, measured), and the Logs rows
are read as tuples, not model objects. The bold, frozen, filtered header,
the column widths and the text cells stay as they were.
record_migration moved or merged a VM's owners without lock_ownership(),
which every other step that moves owners takes; on PostgreSQL a sync of
a third compute with the same UUID could move them at the same time, and
owners moved onto the source record went with its delete.
The name was checked only when the job was queued; recovery counts any
snapshot of that name as the job's, so one taken outside the panel
meanwhile made a failed job succeed. precheck now refuses the name
before any effect.
A drbdadm query that timed out, could not start ssh or failed showed
"None DRBD", as for a VM without DRBD resources. It now shows Unknown,
in the same red badge as a DRBD status that is not OK.
Every connection got a thread before any authentication, and every
signed request a guestfs appliance, with no limit: with remote access
allowed, anyone on that network could make the host run out of threads
or memory. A connection over MAX_CLIENTS (4) now gets an unsigned busy
line and is closed before it gets a thread; the panel shows it as the
gstfsd error.
No view used sensitive_post_parameters, so once ADMINS is set the error
mail of a failing POST would carry the password or key in plain text.
The user create, edit and password views, the password change, the
compute forms, the host secrets, the VM's root password and its console
settings now mark their POST data sensitive.
After the pivot the job asks every compute whether the old file is used;
libvirt has no call timeout, so one stuck host held the job, and a cancel
sent meanwhile held the VM host's control slot, for as long as the host
stayed stuck. The check now waits at most 60 s; a host that has not
answered by then might use the file, which is left behind.
The page checked only that the VM's host answers; a VM undefined with
virsh then failed at its first read and the page was a server error. It
now says the VM is no longer on its host and goes back to the list, whose
sync forgets the record (tr and zh too).
It looked the user's ownership up by the VM's name on its compute; names
are not unique there (a VM renamed outside the panel and a migration can
leave two), which gave a server error or another VM's rights on the page.
The VM and compute claim decorators checked only that the user sees the
VM, took the claim, and left the action's own right to the view inside:
a read-only viewer, or an owner who may not delete the VM, briefly held
the VM's or the whole compute's claim and could make other requests
there busy. They now check the least right first: power for a VM change,
delete for the compute-wide claim (destroy; its other views are
superuser-only).
Deleting a VM with its disks checks every other compute for the files. A
compute that could not be read raised an OperationError whose text already
held that compute's name and libvirt's error, so owners who do not see
hosts got both. It is now a ConnectionFailed, which users who do not see
hosts get as "cannot be reached"; libvirt's text comes from the error
itself, so a connection that never opened no longer reads "None".
The API's VM actions call the web views and only looked at the status
code; the views report a partial result (a disk that was not deleted or
was kept, a template that was not started) as a message for the page
they redirect to. The API reported plain success and left the messages in
the session. Its answer now carries them in "messages".
The API's create serializer leaves mac and nwfilter optional, but the
job's normalize gave them no default: without mac the request ended in an
HTML 500 (KeyError), without nwfilter it was queued and failed in the
worker. Both now default to empty, as the web form sends them.
Every other job re-checks its permission and state in precheck before any
effect; these two checked only when queued. A queued create of a user who
is no longer a superuser, or whose VM name a host took meanwhile, ran; so
did a volume clone whose name a volume made outside the panel took.
Only migrate_instance, inside the job's run, wrote the migration's row:
a migration refused by precheck, cancelled while queued, or ended by
recovery after a worker restart left none. The job now has a log entry
for the ends migrate_instance did not log; run marks those it did.
The internal revert's shut-off check ran in the job's run, after its
intent was recorded, so the refusal went to recovery: the job ended
uncertain and kept the VM's claim until a superuser resolved it. The check
(check_internal_revert) now runs in precheck too, so a page older than the
VM's start, or a direct POST, fails the job with the reason.
The list page synchronised and connected to every compute for every user,
while the poll and the API list read only the hosts of the user's own VMs.
Hosts a user has no VMs on no longer get a listing per page load, which
also took one of their request slots.
A web request refused because this process's requests to the host
(HOST_REQUESTS) all still wait for it raised the same ConnectionFailed as
a host that is down, so the VM page said the host cannot be reached. It
now raises HostBusy; the VM page and the error users who do not see hosts
get say the host is busy and to try again in a moment.
js-cookie served only table_filter.js, which kept a list's search in a
cookie for a day; the cookie went to the server with every request.
The search is now kept in the tab's sessionStorage: it stays across
reloads and pages in that tab, a new tab starts empty, and the server
never sees it. The library goes.
libvirt takes the same disk file or MAC twice in a persistent
definition. A double click or a resent request then attached an
existing volume twice, and a shut-off VM did not start again (QEMU
opens no file twice for writing), or added a second NIC with the MAC
the form had filled in. Attaching a volume that is already a disk of
the VM, live or in its next start, and adding a NIC with a MAC the VM
has are now refused with the reason.
gstfsd_request connected with the 120 seconds it waits for the change:
a host that was down held the request's thread two minutes before its
page said so, a wait no host bound counts. It now connects within
libvirt's CONNECT_TIMEOUT and gives the change its 120 seconds after.
The clone form's checks read the source's NICs from the live and the
persistent definition, and each NIC read the running VM's addresses
again since b2b2386d: a guest agent that is connected but hung held the
request for its timeout once a NIC and definition. The checks now run
in one cached_reads block, as the VM page does.
When the worker stopped after it defined a clone and before it resized
and started it, recovery recorded the clone at the source's size and
shut off, and the job read as a plain success. Recovery now finishes a
defined clone that is shut off as the run does; one that runs went
through the last step, starting, and is left as it is.
clone_data checked the clone's vCPUs and memory with str.isdigit(), which
takes characters such as a superscript two that int() refuses: the user
saw Python's "invalid literal for int()" instead of the refusal. It now
checks with str.isdecimal(), whose characters int() reads.
#394 read each host's states in one call and every domain's XML once,
for the title, memory and vCPUs. A user's list shows only their own VMs,
but it read the XML of every VM on those hosts: a user with two VMs on a
host of 300 made about 301 libvirt calls a list load, where the old
per-VM reads made 6.
get_domain_rows takes the UUIDs the list shows and reads only their XML;
the states still come in one call. The capacity reads every domain, as
before.
607c8381 sent each memory and disk resize back to its own sub-tab, but
the disk view's last redirect, the one after a resize was tried, still
went to #resize, which opens on the CPU sub-tab. Only the live tests
reach it, with a disk to grow: the full live suite failed there on every
connection type.
The live vCPU test built its post from the page's form, and the page now
disables the CPU fields while the VM runs, so the post lacked cur_vcpu
and was refused as not a number before the running VM was. It now posts
both fields, as the direct post it checks.
The VM's keys page (instances:sshkeys) and the cloud-init user data
collected the owners' keys with one query an owner. Both now read them
in one query joined over the VM's owners, ordered owner by owner as
before.
