Any error in the manager loop ended the worker at once with os._exit,
without the drain: a PostgreSQL restart or a SQLite write lock held past
its timeout cut every running migration, block copy and backup the
worker drove, and their jobs went to recovery, some ending uncertain. A
database error now skips that tick and drops the broken connection; the
next tick tries again. A lease lost meanwhile still ends the worker when
renew sees it, and other errors still end it as before.
The worker sends cancels only to jobs its own threads run. After a
worker restart, a migration or live clone that libvirt still ran was
watched by recovery; a cancel then moved the job to cancelling, the page
said so, but nothing told libvirt, and the job ran to its end. Recovery
now sends the cancel once while libvirt still runs the job, and a
migration it aborted ends cancelled rather than failed.
Recovery retried a reconcile that raised every 10 minutes with no end. A
job whose compute was gone for good stayed running ("Waiting for
recovery") and kept its claims: it could not be resolved (only uncertain
jobs can), cancelled, or have its compute deleted. After an hour of
failed reconciles the job ends uncertain with the last error; it keeps
its claims, and an administrator checks the VM and resolves it. A
reconcile that answers restarts the hour.
Nothing removed the Logs page's rows, which every action and every
failed login adds, or the database sessions past their expiry. The job
worker's housekeeping now deletes the rows older than Log Retention
Days, a new setting under Lists and pages (1-3650, 365 by default), and
the expired sessions.
A job's libvirt call to a host that is up but stuck waits as long as
the host does. Measured with the worker's loop: with two jobs stuck on
one host, the other host's jobs ran but their progress was never read
(both progress threads waited on the stuck host); with four stuck, a
new job of another host stayed queued, every slot taken; cancelling
them took all four control threads, so cancels, recovery and
housekeeping stopped for every host.
Now a host's jobs take at most half of the worker's slots (at least
one; a migration counts on both its hosts), and the next queued job of
another host starts instead. A host has at most one progress read and
one control task (a cancel or a recovery inspection) at a time, and
the control tasks of hosts leave one thread to housekeeping.
A job stored its error and its result's warning as English text, much
of it built with the values in it (volume paths, a disk, libvirt's
error), so no catalog could translate it, and the job pages and panel
showed English whatever the language.
A job now stores them as messages: our English text, marked for
translation, and its values; libvirt's own text is a message without
values and is shown as it is. The jobs API and pages put them in the
reader's language. OperationError keeps its text and values apart for
the same reason, so a refusal raised while a job is submitted is shown
translated too. Messages one after another still read as sentences.
Errors stored before are kept as one message each, shown as they were.
The job panel on VM and storage pages wrote its buttons and notes in
English (Cancel, Open, Refresh, Dismiss, Finishing..., Files not
removed, Worker unavailable, Cancel failed) and its state badge as the
state's code, and the jobs API gave the phase untranslated. The panel
now takes its texts and the state labels from the page's settings, and
the API gives the phase in the viewer's language, as it gives the
label; the job pages show that phase.
A job that cannot be cancelled and ran past its time limit had
"(over its time limit)" written into its phase, which then matched no
translation. The phase stays as the job sets it, and the pages say the
job is past its limit.
Request jobs (an upload's chunks) run in gunicorn: their kind is admitted
running with the request as its worker, never claimed or recovered by the
worker, cancelled in the request, and asked by every housekeeping pass to
end its jobs that went idle. The panel can follow a job the page just
started. A job a request runs does not wait for the worker, so it does not
raise the worker-unavailable warning on the Jobs page or in the panel.
Only the request middleware loaded the Settings page values, so in the
worker process app_settings stayed empty: a VM create job failed reading
the default volume format, and the snapshot free-space check fell back to
its defaults. The worker now loads them before each job.
