mirror of
https://github.com/retspen/webvirtcloud
synced 2026-11-08 19:26:48 +00:00
10b46afd36
libvirt has no timeout for opening a connection. Measured in the test setup: an unreachable host blocked qemu+tcp for 135 s and qemu+ssh for 133 s; a daemon that accepts and never answers blocks forever. Gunicorn runs sync workers and the instance list connects to every compute, so one unreachable compute held a worker for minutes per page load. - A connection is opened in a helper thread, at most one per connection at a time. The request waits LIBVIRT_CONNECT_TIMEOUT (default 5 s). An open nobody waits for any more keeps running; until it ends, requests get its error at once, and if it succeeds later the connection is used. A failed open is not repeated for LIBVIRT_RETRY_AFTER (default 30 s). A stuck host holds one thread and does not keep other hosts from connecting. - qemu+ssh runs ssh through vrtManager/libvirt-ssh (the URI's command parameter), which adds ConnectTimeout=10 and BatchMode=yes, so an abandoned ssh process ends and no prompt can hang. The user's ssh configuration (aliases, ProxyJump, keys) still applies. - The close callback acts only for the connection it was registered for: a late callback of a replaced connection dropped its successor. The reason is reported as text. A connection that closed before it was published is an error. - Compute.status is a bool (it returned the exception object, so the "host is down" check in refresh_instance_database never fired); the message is Compute.connection_error. Unit tests cover the timeout, retry delay, late success, single open, a stuck host next to a healthy one and the close callback. Live tests (vrtManager/test_live_connection.py), for TCP and SSH computes of the test setup: an unreachable address (no ssh left after 10 s), an endpoint that never answers, a slow open completing later, a host that goes silent on an established connection (keepalive ends calls within 45 s, the next request reconnects), an ssh_config alias and a ProxyJump host.
67 lines
2 KiB
Python
67 lines
2 KiB
Python
from django.db.models import CharField, IntegerField, Model
|
|
from django.utils.functional import cached_property
|
|
from django.utils.translation import gettext_lazy as _
|
|
from libvirt import virConnect
|
|
|
|
from vrtManager.connection import connection_manager
|
|
from vrtManager.hostdetails import wvmHostDetails
|
|
|
|
|
|
class Compute(Model):
|
|
name = CharField(_("name"), max_length=64, unique=True)
|
|
hostname = CharField(_("hostname"), max_length=64)
|
|
login = CharField(_("login"), max_length=20)
|
|
password = CharField(_("password"), max_length=14, blank=True, null=True)
|
|
details = CharField(_("details"), max_length=64, null=True, blank=True)
|
|
type = IntegerField()
|
|
|
|
@cached_property
|
|
def status(self):
|
|
"""True if WebVirtCloud can connect to the host (within the connect
|
|
timeout; a host that failed recently is not tried again at once)."""
|
|
return isinstance(self.connection, virConnect)
|
|
|
|
@cached_property
|
|
def connection_error(self):
|
|
"""Why the host cannot be reached, or None."""
|
|
return None if self.status else str(self.connection)
|
|
|
|
@cached_property
|
|
def connection(self):
|
|
try:
|
|
return connection_manager.get_connection(
|
|
self.hostname,
|
|
self.login,
|
|
self.password,
|
|
self.type,
|
|
)
|
|
except Exception as e:
|
|
return e
|
|
|
|
@cached_property
|
|
def proxy(self):
|
|
return wvmHostDetails(
|
|
self.hostname,
|
|
self.login,
|
|
self.password,
|
|
self.type,
|
|
)
|
|
|
|
@cached_property
|
|
def cpu_count(self):
|
|
return self.proxy.get_node_info()[3]
|
|
|
|
@cached_property
|
|
def cpu_usage(self):
|
|
return round(self.proxy.get_cpu_usage(diff=False).get('usage'))
|
|
|
|
@cached_property
|
|
def ram_size(self):
|
|
return self.proxy.get_node_info()[2]
|
|
|
|
@cached_property
|
|
def ram_usage(self):
|
|
return self.proxy.get_memory_usage()["percent"]
|
|
|
|
def __str__(self):
|
|
return self.name
|