refresh_instance_database opened the transaction first and then asked
libvirt for the domain list. A host that goes silent on an established
connection keeps that call waiting until keepalive gives up (30 s). On
SQLite, the default database, the open transaction holds a read lock,
so every other write of the app failed meanwhile with "database is
locked", logins (session writes) included. Reproduced in the test
setup: with a relay to a TCP compute frozen after connecting, three
session writes during the 30 s refresh each failed after 5 s.
- The connection check and listAllDomains run before the transaction,
still under the compute lock, so create, delete and migration of the
compute stay excluded as before.
- The transaction only applies the changes. It checks the compute row
first: a compute deleted while libvirt was asked gets no instances
(this ended in a foreign key error before).
A failed listing still leaves the database untouched. With the change,
the same reproduction writes the sessions at once and keeps the
instances. Both new tests fail on the previous code.
libvirt has no timeout for opening a connection. Measured in the test
setup: an unreachable host blocked qemu+tcp for 135 s and qemu+ssh for
133 s; a daemon that accepts and never answers blocks forever. Gunicorn
runs sync workers and the instance list connects to every compute, so
one unreachable compute held a worker for minutes per page load.
- A connection is opened in a helper thread, at most one per
connection at a time. The request waits LIBVIRT_CONNECT_TIMEOUT
(default 5 s). An open nobody waits for any more keeps running; until
it ends, requests get its error at once, and if it succeeds later the
connection is used. A failed open is not repeated for
LIBVIRT_RETRY_AFTER (default 30 s). A stuck host holds one thread and
does not keep other hosts from connecting.
- qemu+ssh runs ssh through vrtManager/libvirt-ssh (the URI's command
parameter), which adds ConnectTimeout=10 and BatchMode=yes, so an
abandoned ssh process ends and no prompt can hang. The user's ssh
configuration (aliases, ProxyJump, keys) still applies.
- The close callback acts only for the connection it was registered
for: a late callback of a replaced connection dropped its successor.
The reason is reported as text. A connection that closed before it
was published is an error.
- Compute.status is a bool (it returned the exception object, so the
"host is down" check in refresh_instance_database never fired); the
message is Compute.connection_error.
Unit tests cover the timeout, retry delay, late success, single open,
a stuck host next to a healthy one and the close callback. Live tests
(vrtManager/test_live_connection.py), for TCP and SSH computes of the
test setup: an unreachable address (no ssh left after 10 s), an endpoint
that never answers, a slow open completing later, a host that goes
silent on an established connection (keepalive ends calls within 45 s,
the next request reconnects), an ssh_config alias and a ProxyJump host.
