websockify logs "Openening socket failed" and calls sys.exit() when its
port is taken, so novncd ended with status 0. Under systemd the unit
then counts as stopped, not failed: Restart=on-failure does not start it
again, systemctl --failed does not list it, and VNC consoles stay down
until someone starts it by hand. novncd now ends with status 1 there.
A request removes its transient claims in a finally. A gunicorn worker
that exits past its timeout ends its other threads' requests too, and an
OOM kill or a container restart ends all of them, so their claims stayed
until they expired 11 minutes later. A VM delete killed this way held
its compute exclusively: every VM action, job and list refresh on that
compute was refused as busy, and nothing in the panel could clear it.
The holder now names the installation by its node id (as the worker
does), which stays the same when the container is recreated, instead of
the hostname. A conflicting claim whose holder is a process of this
installation that no longer runs is removed during admission. A claim
of another installation, or of a PID that is in use again, still waits
for its expiry.
Any error in the manager loop ended the worker at once with os._exit,
without the drain: a PostgreSQL restart or a SQLite write lock held past
its timeout cut every running migration, block copy and backup the
worker drove, and their jobs went to recovery, some ending uncertain. A
database error now skips that tick and drops the broken connection; the
next tick tries again. A lease lost meanwhile still ends the worker when
renew sees it, and other errors still end it as before.
The worker sends cancels only to jobs its own threads run. After a
worker restart, a migration or live clone that libvirt still ran was
watched by recovery; a cancel then moved the job to cancelling, the page
said so, but nothing told libvirt, and the job ran to its end. Recovery
now sends the cancel once while libvirt still runs the job, and a
migration it aborted ends cancelled rather than failed.
migrate_instance sets autostart on the destination (it is per host and
the migration does not carry it) and deletes the NVRAM file libvirt
keeps on the source. A worker that stopped after libvirt moved the VM
skipped both: recovery only recorded the move, so an autostart VM no
longer started after the destination rebooted, and a later VM of the
same name on the source started with these UEFI variables. The job now
records both before the move, and recovery does them after a proven
move; a step that fails is a warning of the job's result.
A shut-off VM with a TPM or disks to copy migrates started paused
(migrate_running), and the job shuts it off afterwards on whichever host
has it. A worker that stopped in between left it running paused, on the
source when the migration did not happen or on the destination when it
did; recovery reported the outcome but left the VM running. The job now
records whether the VM was shut off, and recovery destroys it again
where it ended up.
After a successful copy, recovery defined the clone; a define that
failed raised, and recovery retried it with the same result. One case
is certain: a worker killed after define_clone copied the UEFI
variables but before it defined the VM leaves <clone>_VARS.fd, and every
retry is refused because the volume exists. The job then waited for
recovery with its claims on the source VM and the clone's name. A
libvirt error now ends it uncertain, naming the error and the copied
volumes; an unreachable host still waits.
Recovery retried a reconcile that raised every 10 minutes with no end. A
job whose compute was gone for good stayed running ("Waiting for
recovery") and kept its claims: it could not be resolved (only uncertain
jobs can), cancelled, or have its compute deleted. After an hour of
failed reconciles the job ends uncertain with the last error; it keeps
its claims, and an administrator checks the VM and resolves it. A
reconcile that answers restarts the hour.
The worker checked its VM's state once a second (34dd61c8); a VM started
while its console was open could lose up to a second of its first output
(the BIOS banner, a boot menu) before the stream opened. A VM that is not
running is checked five times a second again; one that runs stays at once
a second.
Django logs every 4xx as a django.request WARNING; each 404 of a scanner
or an old link was a line in the app log, burying the ones that matter.
404s are left out; 403s and errors stay (a refused page also has a Logs
row).
get_user_totp_device looked for the user's TOTP device and made one if
there was none; two requests at once could both make one, and both
devices' codes then logged in. It now does so under a lock on the user's
row.
A double click made two tokens: the second key was shown, the first
never, yet it stayed valid until it expired and took a place of the ten.
A user's token names are now unique, checked under a lock on the user's
row, which also keeps two forms sent at once from passing the limit
together. tr and zh too.
A CD-ROM form sent twice (a double click, a resent request) added a
second drive with the same ISO. The form is refused when the ISO is in a
CD-ROM of the VM's running or persistent definition; an empty drive may
still be added more than once. tr and zh too.
The workbook kept every cell as an object: a 100 000-row Logs download
took some 400 MB in a web worker. openpyxl's write-only mode writes the
rows as they come (85 MB for the same file, measured), and the Logs rows
are read as tuples, not model objects. The bold, frozen, filtered header,
the column widths and the text cells stay as they were.
Each poll called info(), then blockStats() for every disk and
interfaceStats() for every NIC: nine calls for a VM with four disks and
two NICs, every 5 s for each open Stats tab. domainListGetStats now gives
the CPU time and every disk's and NIC's counters at once; a counter the
host does not report is -1, as before.
Its worker asked libvirt for the VM's state five times a second, up to
100 calls a second with 20 consoles open, only to notice a start or a
stop. Once a second is enough: output and keys do not wait for it.
consolecallback read the VM's record with .get(): one deleted between
socketiod's accept and the worker's start ended with a Python traceback
in the user's terminal, which is the worker's stderr. It now says the VM
is no longer in the panel.
record_migration moved or merged a VM's owners without lock_ownership(),
which every other step that moves owners takes; on PostgreSQL a sync of
a third compute with the same UUID could move them at the same time, and
owners moved onto the source record went with its delete.
The name was checked only when the job was queued; recovery counts any
snapshot of that name as the job's, so one taken outside the panel
meanwhile made a failed job succeed. precheck now refuses the name
before any effect.
A drbdadm query that timed out, could not start ssh or failed showed
"None DRBD", as for a VM without DRBD resources. It now shows Unknown,
in the same red badge as a DRBD status that is not OK.
Every connection got a thread before any authentication, and every
signed request a guestfs appliance, with no limit: with remote access
allowed, anyone on that network could make the host run out of threads
or memory. A connection over MAX_CLIENTS (4) now gets an unsigned busy
line and is closed before it gets a thread; the panel shows it as the
gstfsd error.
No view used sensitive_post_parameters, so once ADMINS is set the error
mail of a failing POST would carry the password or key in plain text.
The user create, edit and password views, the password change, the
compute forms, the host secrets, the VM's root password and its console
settings now mark their POST data sensitive.
websockify's socket() connects without a timeout, so a console address
that does not answer held the console's process for the OS's TCP timeout
(some 2 minutes) before the page heard of it. novncd now connects with
CONNECT_TIMEOUT, then waits on the open connection as before.
The Lost OTP? page, open without a login, and the admin's Email OTP QR
code mailed the QR code itself: the TOTP secret then stayed in the
mailbox and on every server the mail passed. The mail now carries a
signed link, good for an hour, to a page that shows the QR code (not
cached); the page shows nothing for an old or altered link or a user
without a device. tr and zh too.
gstfsd read its port from the environment while the panel has 16510 as a
constant, so a host that set GSTFSD_PORT could no longer be reached. The
documentation already gives 16510 everywhere; the variable now says it
exists for the tests.
The VNC page's token travels in the websocket URL, so nginx's access log
and anyone who saw the URL had it for 120 s, and it opened the console as
a bearer token. novncd now also wants the session cookie of the token's
user. Its refusals were a warning without a user or an address: they now
name the client address (nginx sends X-Real-IP to novncd too) and leave
a Logs row "VNC console refused: reason".
After the pivot the job asks every compute whether the old file is used;
libvirt has no call timeout, so one stuck host held the job, and a cancel
sent meanwhile held the VM host's control slot, for as long as the host
stayed stuck. The check now waits at most 60 s; a host that has not
answered by then might use the file, which is left behind.
The page checked only that the VM's host answers; a VM undefined with
virsh then failed at its first read and the page was a server error. It
now says the VM is no longer on its host and goes back to the list, whose
sync forgets the record (tr and zh too).
It looked the user's ownership up by the VM's name on its compute; names
are not unique there (a VM renamed outside the panel and a migration can
leave two), which gave a server error or another VM's rights on the page.
The VM and compute claim decorators checked only that the user sees the
VM, took the claim, and left the action's own right to the view inside:
a read-only viewer, or an owner who may not delete the VM, briefly held
the VM's or the whole compute's claim and could make other requests
there busy. They now check the least right first: power for a VM change,
delete for the compute-wide claim (destroy; its other views are
superuser-only).
The worker's checks refuse with PermissionDenied or Http404, which carry
no text: the job failed with an empty error and its Logs row gave no
reason. They now read "You may no longer do this." and "The virtual
machine is no longer in the panel." (tr and zh too).
Deleting a VM with its disks checks every other compute for the files. A
compute that could not be read raised an OperationError whose text already
held that compute's name and libvirt's error, so owners who do not see
hosts got both. It is now a ConnectionFailed, which users who do not see
hosts get as "cannot be reached"; libvirt's text comes from the error
itself, so a connection that never opened no longer reads "None".
The API's VM actions call the web views and only looked at the status
code; the views report a partial result (a disk that was not deleted or
was kept, a template that was not started) as a message for the page
they redirect to. The API reported plain success and left the messages in
the session. Its answer now carries them in "messages".
The API's create serializer leaves mac and nwfilter optional, but the
job's normalize gave them no default: without mac the request ended in an
HTML 500 (KeyError), without nwfilter it was queued and failed in the
worker. Both now default to empty, as the web form sends them.
Every other job re-checks its permission and state in precheck before any
effect; these two checked only when queued. A queued create of a user who
is no longer a superuser, or whose VM name a host took meanwhile, ran; so
did a volume clone whose name a volume made outside the panel took.
Only migrate_instance, inside the job's run, wrote the migration's row:
a migration refused by precheck, cancelled while queued, or ended by
recovery after a worker restart left none. The job now has a log entry
for the ends migrate_instance did not log; run marks those it did.
The internal revert's shut-off check ran in the job's run, after its
intent was recorded, so the refusal went to recovery: the job ended
uncertain and kept the VM's claim until a superuser resolved it. The check
(check_internal_revert) now runs in precheck too, so a page older than the
VM's start, or a direct POST, fails the job with the reason.
The list page synchronised and connected to every compute for every user,
while the poll and the API list read only the hosts of the user's own VMs.
Hosts a user has no VMs on no longer get a listing per page load, which
also took one of their request slots.
A web request refused because this process's requests to the host
(HOST_REQUESTS) all still wait for it raised the same ConnectionFailed as
a host that is down, so the VM page said the host cannot be reached. It
now raises HostBusy; the VM page and the error users who do not see hosts
get say the host is busy and to try again in a moment.
The token now names the console type it was made for: novncd accepts only
a VNC page's token and socketiod only a serial page's, so a VNC page's
token no longer opens the serial console without the VNC password.
A VNC session re-checks every 10 s, as the serial one did, that its user
may still open the console; a deactivated user or a removed owner loses
it. The check (may_still_open) moves to console.tokens for both daemons.
libvirt's default handler printed each error on stderr without saying
where it came from: in Docker those lines went to docker logs, and a
VM page read of a shut-off VM left one there. Every process that talks to
libvirt imports vrtManager.connection, which now registers a silent
handler; the errors still reach their callers as exceptions. The
console callback's own registration is dropped.
Each 401 or 403 answered to a signed-in user or to a request with
credentials (an API token) writes "Access denied (status): method path"
with the user and the address. Anonymous refusals are left out, so a
scanner cannot fill the table.
A run of failed opens is logged once, at its first failure, and the next
successful open ends it; a close callback and a request refused because the
host's requests still wait are logged each time. The lines go to the app log
of the process (web or job worker).
A request gets nginx's $request_id (or a new one when it comes without a
valid one) and sends it back in X-Request-ID. The app log's lines carry
the logger, process and thread, the request ID, the signed-in user and the
client address; gunicorn's access lines carry the client address from
X-Real-IP and the same request ID, so the two logs can be joined.
js-cookie served only table_filter.js, which kept a list's search in a
cookie for a day; the cookie went to the server with every request.
The search is now kept in the tab's sessionStorage: it stays across
reloads and pages in that tab, a new tab starts empty, and the server
never sees it. The library goes.
