A request gets nginx's $request_id (or a new one when it comes without a
valid one) and sends it back in X-Request-ID. The app log's lines carry
the logger, process and thread, the request ID, the signed-in user and the
client address; gunicorn's access lines carry the client address from
X-Real-IP and the same request ID, so the two logs can be joined.
A host that is up but does not answer (a hung NFS mount, for one) keeps
a libvirt call waiting: libvirt's keepalive does not end it, since the
daemon still answers keepalives. Each such request held a whole sync
worker, and once they held every worker the panel stopped answering,
pages without libvirt too; F5 added another.
gunicorn now runs threaded workers, 4 threads each, and a worker takes
no more connections than it has threads, so one whose threads are all
busy leaves new requests to the others. A threaded worker keeps
reporting in while a thread waits, so gunicorn's timeout no longer ends
the request: the worker now exits itself when a request runs past the
timeout (600 s), and gunicorn starts a new one. Short claims expire on
that bound.
The address search runs one at a time in a process, as it did with one
request per worker.
gunicorn 26 opens a control socket in the service user's home, which
www-data cannot create, and logs an error at every start. The panel
does not use it.
