Ten classes ran on TestCase or TransactionTestCase although they read
files, templates or scripts only. SimpleTestCase skips the transaction
and refuses a database query, so a test that starts to use the database
says so. LoggingConfigTestCase also removes the temporary directory it
makes.
Any error in the manager loop ended the worker at once with os._exit,
without the drain: a PostgreSQL restart or a SQLite write lock held past
its timeout cut every running migration, block copy and backup the
worker drove, and their jobs went to recovery, some ending uncertain. A
database error now skips that tick and drops the broken connection; the
next tick tries again. A lease lost meanwhile still ends the worker when
renew sees it, and other errors still end it as before.
Nothing removed the Logs page's rows, which every action and every
failed login adds, or the database sessions past their expiry. The job
worker's housekeeping now deletes the rows older than Log Retention
Days, a new setting under Lists and pages (1-3650, 365 by default), and
the expired sessions.
A job's libvirt call to a host that is up but stuck waits as long as
the host does. Measured with the worker's loop: with two jobs stuck on
one host, the other host's jobs ran but their progress was never read
(both progress threads waited on the stuck host); with four stuck, a
new job of another host stayed queued, every slot taken; cancelling
them took all four control threads, so cancels, recovery and
housekeeping stopped for every host.
Now a host's jobs take at most half of the worker's slots (at least
one; a migration counts on both its hosts), and the next queued job of
another host starts instead. A host has at most one progress read and
one control task (a cancel or a recovery inspection) at a time, and
the control tasks of hosts leave one thread to housekeeping.
A job stored its error and its result's warning as English text, much
of it built with the values in it (volume paths, a disk, libvirt's
error), so no catalog could translate it, and the job pages and panel
showed English whatever the language.
A job now stores them as messages: our English text, marked for
translation, and its values; libvirt's own text is a message without
values and is shown as it is. The jobs API and pages put them in the
reader's language. OperationError keeps its text and values apart for
the same reason, so a refusal raised while a job is submitted is shown
translated too. Messages one after another still read as sentences.
Errors stored before are kept as one message each, shown as they were.
Request jobs (an upload's chunks) run in gunicorn: their kind is admitted
running with the request as its worker, never claimed or recovered by the
worker, cancelled in the request, and asked by every housekeeping pass to
end its jobs that went idle. The panel can follow a job the page just
started. A job a request runs does not wait for the worker, so it does not
raise the worker-unavailable warning on the Jobs page or in the panel.
Only the request middleware loaded the Settings page values, so in the
worker process app_settings stayed empty: a VM create job failed reading
the default volume format, and the snapshot free-space check fell back to
its defaults. The worker now loads them before each job.
