- console settings were split over four files; who sees the form, the
listen address, and the settings themselves are test_console_settings,
and who may open a console is test_console_access;
- the three template files are test_templates;
- the quota lock test joins test_quota;
- accounts' test_security_fixes held two topics: the SSH key tests are
test_ssh_keys, the owner-row delete redirect is in test_views;
- instances' test_security_mutations, who may change a VM, is
test_permissions.
The suite runs the same 2097 tests.
Tests that build vrtManager objects and touch no view, form or model of
an app lived in that app; vrtManager/tests already holds such tests, so
the convention was split. They move to vrtManager/tests: the firmware,
CPU mode and PCIe port definitions of a new VM (test_create), USB and PCI
host devices, the console listener, host CPU reads, interface reads,
deletes and XML, network filters, PCI passthrough rules, node devices,
network changes, addresses and leases, pool XML, secrets, and the live
tests that use vrtManager only (disk resize, ISO, snapshots, XML, live
clone). livetest.py, the live-test helper six apps import, and the
network test driver move with them.
Others go to their app: the quota with a host down to instances, the
401/403 Logs rows (webvirtcloud's middleware) to webvirtcloud, the VM
page's tab script test to instances, novncd's own tests out of the
console views tests, the compute lock and input validation tests out of
the computes views tests, the interface views out of the forms tests.
instances/tests/test_views.py, entirely live, is test_live_views.py.
Test modules imported fixtures from each other; shared ones are now in
vrtManager/tests/fakes.py (migration fakes, node devices, IOMMU groups,
leases, a NAT network, a Ceph secret), interfaces/tests/fakes.py,
console/tests/daemons.py (novncd loaded once), livetest.wait_for and
instances/tests/create_page.BODY. Every live test checks
livetest.enabled(). The suite runs the same 2097 tests.
conf/test_install_lib.sh tested conf/install-lib.sh, but no CI step or
test ran it. It moves to webvirtcloud/tests/sh/install_lib.test.sh, takes
the library's path as its argument, and test_install_lib runs it in bash,
as the JS tests run in node.
Five tests only acquired and released the lock and asserted nothing.
They now check, from another thread, that a reader lets readers in and
keeps writers out, that a writer keeps everyone out, and that a nested
or upgraded lock is held until its last release.
- 25 vrtManager test modules configured settings themselves when Django
had none (MAC_OUI, and in test_network_ip a fake django module): under
manage.py test settings are always configured, so it never ran.
- webvirtcloud/tests/test_common_tags.py tested no tag: it fetched an
unknown URL, as test_error_pages and test_logging already do.
- .gitignore covers the parallel runs' test_db_N.sqlite3 files too.
Ten classes ran on TestCase or TransactionTestCase although they read
files, templates or scripts only. SimpleTestCase skips the transaction
and refuses a database query, so a test that starts to use the database
says so. LoggingConfigTestCase also removes the temporary directory it
makes.
- create_admin had a branch for `"test" in sys.argv` that gave the admin
the password "admin"; 17 places signed in with it, and failed silently
with ADMIN_PASSWORD set in the environment. The tests now make their
own superuser and force_login it, and the shipped code has no test
branch: the test runner gives migrate a password (also when
ADMIN_PASSWORD is set but empty), so no data/admin_password is written
into the checkout.
- AccountsTestCase builds its rows in setUpTestData instead of
setUpClass and deleting them by hand.
- accounts tests set OTP_ENABLED with override_settings instead of
assigning settings (one assignment was never undone, so later tests in
the process ran with OTP on), and undo their app_settings change. The
OTP login class reloads the URLs again only after the setting is off.
- computes/tests/test_views.py reached the machine's own libvirt
(localhost over the socket) when TEST_LIBVIRT_HOST was unset; its live
tests now run only with TEST_LIBVIRT_HOST, like the other live tests,
as the README says.
0001_initial to 0024_log_retention_setting are mostly RunPython steps
that added, renamed, re-chose and described setting rows since 2020. A
squashed migration now creates the table in its final shape and inserts
the 38 rows those steps leave; it replaces all 24, so a new database
runs it alone. The 24 stay beside it, as Django's squashing expects
until every installation has applied them: a database that applied
them all records the squashed one without running anything, and one
that stopped between them (an older release, or an upstream
installation) applies the rest of them first.
Checked:
- on a new database its rows equal the old chain's (name, key, value,
choices, description of every row);
- on a copy of a running panel's database (all 24 applied) migrate
plans nothing for appsettings;
- a database migrated by the old code to 0023 applies 0024 and then
records the squashed migration, with the same 38 rows;
- makemigrations finds no change.
It inserts its rows on the database being migrated, as the replaced steps
do. The tests of the replaced steps stay, since an installation that stopped
between them still runs them; the two that checked a few rows' columns
became one that checks every row's columns and description, and a value is
one of its choices.
Networks, Interfaces, Storages, Storage, Volume, MigrateInstance and
CreateInstance were unmanaged models with no table, there only so that
ModelSerializers and the new VM ModelForm could take their fields. They
made content types and migrations, and their DO_NOTHING foreign keys
added reverse accessors that meant nothing (Compute.host,
Compute.target_host, Instance.source_host).
The serializers are plain Serializers and NewVMForm a Form, field for
field as the models gave them (types, lengths, required, defaults, the
minimum of 1 for vCPUs, memory and disk size; the form's labels are
still derived from the field names). Compared field by field before and
after, the API schema changes only where the models were wrong:
- no int64 bounds on the listing fields;
- the interface fields lose the titles that named the wrong column
(type was "Status", state "Device", mac "Forward");
- a pool's volumes is a list of objects (it was described as an
integer) and a volume's allocation an integer (it was untyped).
One migration per app deletes the models; none touches a table.
It created the group with objects.create and had no reverse: it failed
when the group existed and could not be unapplied. It now uses
get_or_create, and its reverse does nothing.
TEMPLATE_DEBUG has been unused since Django 1.10 and debug toolbar no
longer reads INTERCEPT_REDIRECTS; DebugToolbarMiddleware goes first, as
its documentation asks, instead of after django-axes.
instances/forms.py and instances/models.py imported values from the
webvirtcloud.settings module itself, which bypasses DJANGO_SETTINGS_MODULE
(settings-dev, test overrides) and runs the module a second time. They
now read django.conf.settings. webvirtcloud/urls.py also loses its
fallback that set REST_FRAMEWORK's schema class at import for settings.py
copies without it: the template sets it, and an older settings.py is
rebuilt from the template when updating.
The operations app set PRAGMA journal_mode=WAL from a connection_created
receiver because Django's SQLite OPTIONS had no init command; Django 5.1
added init_command, so the template sets it there and the receiver goes.
The mode is stored in the database file, so existing databases keep it.
LOCALE_PATHS was the relative "locale/": a manage.py or cron run outside
the checkout found no catalogs and fell back to English. It is now under
BASE_DIR. Reading data/secret_key also catches only OSError instead of
any exception.
AUTH_PASSWORD_VALIDATORS was unset, so the user forms and the password
change accepted any password, a one-letter one included. The template
now sets Django's four standard validators: at least 8 characters, not
a common password, not only digits, not like the user's name or e-mail.
The tests that set weak passwords use ones the validators accept.
No RemoteUserBackend is configured, so the middleware never logged
anyone in: it only ran on every request and read as if the panel trusted
a REMOTE_USER header.
The root password hash for a shut-off VM used crypt_r where installed
(Python 3.13) and fell back to the stdlib crypt, which Python 3.11 and
3.12 deprecate: `python -W error::DeprecationWarning manage.py check`
failed there. crypt-r (the same module, it requires Python 3.11) is now
installed from 3.11, and the module is chosen by the Python version;
3.10's crypt is not deprecated.
restart_nginx only logged "systemctl enable/restart nginx.service": both
calls had been commented out since the installer was added (2020). On
RHEL and openSUSE nginx is not enabled after its install, so the panel
was not served until someone started it; on Debian and Ubuntu nginx
kept the default site until a reload. The installer now checks the
config, enables nginx and restarts it, and stops if that fails.
- noVNC's own web client (app/ except the bell sounds, vnc.html,
vnc_lite.html, defaults.json, mandatory.json): the console imports
only noVNC's core, and novncd serves no web pages. AUTHORS, LICENSE.txt
and package.json (the version) stay.
- 24 cached icons no {% bs_icon %} asks for; the icon cache test now
also fails on an icon nothing uses.
- Ace's ext-textarea.js, which nothing loads.
- instances/fixtures/initial_data.json, for an app ("create") that no
longer exists; migration 0006 seeds the flavors.
- webvirtcloud/asgi.py: gunicorn runs the WSGI application only.
- instances/templatetags/tags_active.py: a copy of common_tags'
class_active that no template loads.
- .github/linters/.hadolint.yaml, from super-linter, which no workflow
runs anymore.
Each was checked with git grep across every tracked file (templates, JS,
shell, unit files and the extensionless console daemons) and for
dynamic use (URL names, template names, registries):
- vrtManager: host_is_up, is_kvm_supported, get_emulators,
get_file_extensions, both get_network_forward, get_snapshots,
get_storages_images, get_os_type, get_host_arch, the get_pretty_* trio
and, in util, is_kvm_available, randomMAC, get_max_vcpus, compareMAC,
pretty_mem, pretty_bytes and validate_uuid; with them the port
constants only host_is_up read. Some were reached only by their tests,
which go too.
- The flavor edit page: nothing linked to it; flavors are created and
deleted on the Flavors page and edited through the API.
- computes.validate_name, UserForm.identity_has_errors, the has_perm
template tag, the ALLOW_EMPTY_PASSWORD setting nothing reads,
hidePleaseWaitDialog, default_app_config (ignored since Django 4.1)
and the Python < 3.6 lines of secret_generator.py.
The delete ran in the request: undefine, then the VM's record, then its
disk files. A request that died in between (a gunicorn worker past its
timeout, OOM, a restart), or a database write that failed after the
undefine, left a record that later showed as removed outside the panel,
and the disk files the user asked to delete were never deleted.
Deleting a VM is now an instance.delete job. It holds the compute
exclusively, as the request did, so no other VM attaches a file it is
about to delete. The job chooses the files before it stops the VM (a host
that cannot be checked still stops nothing) and records them first. It
then stops and undefines the VM, removes its record, and deletes the
files, each with its own Logs row. Recovery decides from libvirt: a VM
still on its host means the delete failed; a VM gone means libvirt did
it, so recovery removes the record and deletes the recorded files that
are still there.
The page queues the job and returns to the VM page, where the job panel
follows it and opens the VM list when it ends; disks that were kept are
the result's warning. The API's delete answers 202 with the job and
needs a submission_key. backing_path moves to wvmConnect, as recovery
has no domain to ask.
The update steps wrote the copy straight to $b/db.sqlite3: the shell
creates the file before the copy runs, so a copy that failed left an
empty or partial db.sqlite3 next to the commit, which looks like a good
backup. Restoring from it put an empty database in place, and the panel
came back with no users, computes or VMs. The copy is now written to
db.sqlite3.part and renamed when it succeeds, in the Docker and the
installer steps.
A panel that takes SECRET_KEY from its environment has no data/secret_key,
so its backup has none. The Docker moving steps copied it
unconditionally: the copy failed after the database had already been
replaced, and the chain stopped before node_id, the SSH keys and `up -d`.
The copy is now skipped when the backup has no key, and the section
says to set the same value on the new machine. It also says to stop the
old panel first when both use one PostgreSQL database: the moved worker
id would let both workers take each other's lease and jobs.
libvirt reads a TLS compute's client certificate from the panel user's
~/.pki/libvirt. In Docker that was /var/www/.pki inside the container,
so `docker compose up -d --build` dropped it and every TLS compute
stopped connecting; no backup held it on either install type, although
"Moving to a new machine" says the backup carries what the computes
accept, and the README never said where the certificates go.
Docker keeps /var/www/.pki in a webvirtcloud-pki volume, owned by
www-data and private like ~/.ssh; the backup copies ~/.pki into pki/;
the moving steps put it back; and "Adding a Compute Node" says where the
files go.
The container's health check only fetched the login page, through nginx
and gunicorn. A job worker, novncd or socketiod that kept failing (runit
starts it again each second) left the container healthy while jobs
waited and consoles failed. The check now also asks runit for each
service and fails when one is down or has run for less than five
seconds.
websockify logs "Openening socket failed" and calls sys.exit() when its
port is taken, so novncd ended with status 0. Under systemd the unit
then counts as stopped, not failed: Restart=on-failure does not start it
again, systemctl --failed does not list it, and VNC consoles stay down
until someone starts it by hand. novncd now ends with status 1 there.
A request removes its transient claims in a finally. A gunicorn worker
that exits past its timeout ends its other threads' requests too, and an
OOM kill or a container restart ends all of them, so their claims stayed
until they expired 11 minutes later. A VM delete killed this way held
its compute exclusively: every VM action, job and list refresh on that
compute was refused as busy, and nothing in the panel could clear it.
The holder now names the installation by its node id (as the worker
does), which stays the same when the container is recreated, instead of
the hostname. A conflicting claim whose holder is a process of this
installation that no longer runs is removed during admission. A claim
of another installation, or of a PID that is in use again, still waits
for its expiry.
Any error in the manager loop ended the worker at once with os._exit,
without the drain: a PostgreSQL restart or a SQLite write lock held past
its timeout cut every running migration, block copy and backup the
worker drove, and their jobs went to recovery, some ending uncertain. A
database error now skips that tick and drops the broken connection; the
next tick tries again. A lease lost meanwhile still ends the worker when
renew sees it, and other errors still end it as before.
The worker sends cancels only to jobs its own threads run. After a
worker restart, a migration or live clone that libvirt still ran was
watched by recovery; a cancel then moved the job to cancelling, the page
said so, but nothing told libvirt, and the job ran to its end. Recovery
now sends the cancel once while libvirt still runs the job, and a
migration it aborted ends cancelled rather than failed.
migrate_instance sets autostart on the destination (it is per host and
the migration does not carry it) and deletes the NVRAM file libvirt
keeps on the source. A worker that stopped after libvirt moved the VM
skipped both: recovery only recorded the move, so an autostart VM no
longer started after the destination rebooted, and a later VM of the
same name on the source started with these UEFI variables. The job now
records both before the move, and recovery does them after a proven
move; a step that fails is a warning of the job's result.
A shut-off VM with a TPM or disks to copy migrates started paused
(migrate_running), and the job shuts it off afterwards on whichever host
has it. A worker that stopped in between left it running paused, on the
source when the migration did not happen or on the destination when it
did; recovery reported the outcome but left the VM running. The job now
records whether the VM was shut off, and recovery destroys it again
where it ended up.
After a successful copy, recovery defined the clone; a define that
failed raised, and recovery retried it with the same result. One case
is certain: a worker killed after define_clone copied the UEFI
variables but before it defined the VM leaves <clone>_VARS.fd, and every
retry is refused because the volume exists. The job then waited for
recovery with its claims on the source VM and the clone's name. A
libvirt error now ends it uncertain, naming the error and the copied
volumes; an unreachable host still waits.
Recovery retried a reconcile that raised every 10 minutes with no end. A
job whose compute was gone for good stayed running ("Waiting for
recovery") and kept its claims: it could not be resolved (only uncertain
jobs can), cancelled, or have its compute deleted. After an hour of
failed reconciles the job ends uncertain with the last error; it keeps
its claims, and an administrator checks the VM and resolves it. A
reconcile that answers restarts the hour.
The worker checked its VM's state once a second (34dd61c8); a VM started
while its console was open could lose up to a second of its first output
(the BIOS banner, a boot menu) before the stream opened. A VM that is not
running is checked five times a second again; one that runs stays at once
a second.
Django logs every 4xx as a django.request WARNING; each 404 of a scanner
or an old link was a line in the app log, burying the ones that matter.
404s are left out; 403s and errors stay (a refused page also has a Logs
row).
get_user_totp_device looked for the user's TOTP device and made one if
there was none; two requests at once could both make one, and both
devices' codes then logged in. It now does so under a lock on the user's
row.
A double click made two tokens: the second key was shown, the first
never, yet it stayed valid until it expired and took a place of the ten.
A user's token names are now unique, checked under a lock on the user's
row, which also keeps two forms sent at once from passing the limit
together. tr and zh too.
A CD-ROM form sent twice (a double click, a resent request) added a
second drive with the same ISO. The form is refused when the ISO is in a
CD-ROM of the VM's running or persistent definition; an empty drive may
still be added more than once. tr and zh too.
The workbook kept every cell as an object: a 100 000-row Logs download
took some 400 MB in a web worker. openpyxl's write-only mode writes the
rows as they come (85 MB for the same file, measured), and the Logs rows
are read as tuples, not model objects. The bold, frozen, filtered header,
the column widths and the text cells stay as they were.
Each poll called info(), then blockStats() for every disk and
interfaceStats() for every NIC: nine calls for a VM with four disks and
two NICs, every 5 s for each open Stats tab. domainListGetStats now gives
the CPU time and every disk's and NIC's counters at once; a counter the
host does not report is -1, as before.
Its worker asked libvirt for the VM's state five times a second, up to
100 calls a second with 20 consoles open, only to notice a start or a
stop. Once a second is enough: output and keys do not wait for it.
consolecallback read the VM's record with .get(): one deleted between
socketiod's accept and the worker's start ended with a Python traceback
in the user's terminal, which is the worker's stderr. It now says the VM
is no longer in the panel.
record_migration moved or merged a VM's owners without lock_ownership(),
which every other step that moves owners takes; on PostgreSQL a sync of
a third compute with the same UUID could move them at the same time, and
owners moved onto the source record went with its delete.
The name was checked only when the job was queued; recovery counts any
snapshot of that name as the job's, so one taken outside the panel
meanwhile made a failed job succeed. precheck now refuses the name
before any effect.
