A job's libvirt call to a host that is up but stuck waits as long as
the host does. Measured with the worker's loop: with two jobs stuck on
one host, the other host's jobs ran but their progress was never read
(both progress threads waited on the stuck host); with four stuck, a
new job of another host stayed queued, every slot taken; cancelling
them took all four control threads, so cancels, recovery and
housekeeping stopped for every host.
Now a host's jobs take at most half of the worker's slots (at least
one; a migration counts on both its hosts), and the next queued job of
another host starts instead. A host has at most one progress read and
one control task (a cancel or a recovery inspection) at a time, and
the control tasks of hosts leave one thread to housekeeping.
A job stored its error and its result's warning as English text, much
of it built with the values in it (volume paths, a disk, libvirt's
error), so no catalog could translate it, and the job pages and panel
showed English whatever the language.
A job now stores them as messages: our English text, marked for
translation, and its values; libvirt's own text is a message without
values and is shown as it is. The jobs API and pages put them in the
reader's language. OperationError keeps its text and values apart for
the same reason, so a refusal raised while a job is submitted is shown
translated too. Messages one after another still read as sentences.
Errors stored before are kept as one message each, shown as they were.
Request jobs (an upload's chunks) run in gunicorn: their kind is admitted
running with the request as its worker, never claimed or recovered by the
worker, cancelled in the request, and asked by every housekeeping pass to
end its jobs that went idle. The panel can follow a job the page just
started. A job a request runs does not wait for the worker, so it does not
raise the worker-unavailable warning on the Jobs page or in the panel.
Only the request middleware loaded the Settings page values, so in the
worker process app_settings stayed empty: a VM create job failed reading
the default volume format, and the snapshot free-space check fell back to
its defaults. The worker now loads them before each job.
