mirror of
https://github.com/retspen/webvirtcloud
synced 2026-11-10 12:16:49 +00:00
fix(vrtManager): libvirt connections open within a time limit
libvirt has no timeout for opening a connection. Measured in the test setup: an unreachable host blocked qemu+tcp for 135 s and qemu+ssh for 133 s; a daemon that accepts and never answers blocks forever. Gunicorn runs sync workers and the instance list connects to every compute, so one unreachable compute held a worker for minutes per page load. - A connection is opened in a helper thread, at most one per connection at a time. The request waits LIBVIRT_CONNECT_TIMEOUT (default 5 s). An open nobody waits for any more keeps running; until it ends, requests get its error at once, and if it succeeds later the connection is used. A failed open is not repeated for LIBVIRT_RETRY_AFTER (default 30 s). A stuck host holds one thread and does not keep other hosts from connecting. - qemu+ssh runs ssh through vrtManager/libvirt-ssh (the URI's command parameter), which adds ConnectTimeout=10 and BatchMode=yes, so an abandoned ssh process ends and no prompt can hang. The user's ssh configuration (aliases, ProxyJump, keys) still applies. - The close callback acts only for the connection it was registered for: a late callback of a replaced connection dropped its successor. The reason is reported as text. A connection that closed before it was published is an error. - Compute.status is a bool (it returned the exception object, so the "host is down" check in refresh_instance_database never fired); the message is Compute.connection_error. Unit tests cover the timeout, retry delay, late success, single open, a stuck host next to a healthy one and the close callback. Live tests (vrtManager/test_live_connection.py), for TCP and SSH computes of the test setup: an unreachable address (no ssh left after 10 s), an endpoint that never answers, a slow open completing later, a host that goes silent on an established connection (keepalive ends calls within 45 s, the next request reconnects), an ssh_config alias and a ProxyJump host.
This commit is contained in:
parent
99eadaab31
commit
10b46afd36
9 changed files with 517 additions and 86 deletions
12
vrtManager/libvirt-ssh
Executable file
12
vrtManager/libvirt-ssh
Executable file
|
|
@ -0,0 +1,12 @@
|
|||
#!/bin/sh
|
||||
# ssh for libvirt qemu+ssh connections (the URI's "command" parameter).
|
||||
# libvirt passes no connect timeout, so an unreachable host would keep this
|
||||
# ssh process waiting for the kernel's TCP timeout. BatchMode makes ssh fail
|
||||
# instead of waiting for a password or host key prompt nobody can answer.
|
||||
# The timeout is fixed: libvirt passes this script no environment or extra
|
||||
# arguments. It bounds only ssh's TCP connect; WebVirtCloud stops waiting
|
||||
# after LIBVIRT_CONNECT_TIMEOUT on its own, so an abandoned ssh lives at most
|
||||
# this long.
|
||||
# Everything else comes from the user's ssh configuration (aliases,
|
||||
# ProxyJump, keys).
|
||||
exec ssh -o ConnectTimeout=10 -o BatchMode=yes "$@"
|
||||
Loading…
Add table
Add a link
Reference in a new issue