A host that is up but does not answer (a hung NFS mount, for one) keeps
a libvirt call waiting: libvirt's keepalive does not end it, since the
daemon still answers keepalives. Each such request held a whole sync
worker, and once they held every worker the panel stopped answering,
pages without libvirt too; F5 added another.
gunicorn now runs threaded workers, 4 threads each, and a worker takes
no more connections than it has threads, so one whose threads are all
busy leaves new requests to the others. A threaded worker keeps
reporting in while a thread waits, so gunicorn's timeout no longer ends
the request: the worker now exits itself when a request runs past the
timeout (600 s), and gunicorn starts a new one. Short claims expire on
that bound.
The address search runs one at a time in a process, as it did with one
request per worker.
The IP search and the VM report read each running VM's addresses and
OS from its guest agent with interfaceAddresses() and guestInfo(),
which wait for a hung agent indefinitely. The search answered by its
deadline, but the thread stayed blocked, counted against the search
thread limit and held up the worker's exit.
They now use the 10 s agent command the VM page uses, shared as
vrtManager.instance.agent_command() and agent_interfaces(). With the
agent frozen, a search answers in 5 s from the leases and the ARP
table, counts the VM unread, and its thread ends about 5 s later.
A paused guest cannot answer its agent, so the address search and the VM
report asked it nothing, but counted the VM neither as without an agent
nor as not read: the coverage line read as if every address had been
seen. It now counts among the agents not read, and the line names the
paused case.
The "Find IP" dialog takes a MAC address (aa:bb:..., aa-bb-... or
aabb...) as well as an IP address or a network. A MAC finds the VMs with
a network card of that MAC, shut-off ones too (from their definitions),
with that card's addresses when the VM runs: for a MAC seen on a switch
port or in an external DHCP server's log. A MAC on more than one VM is
marked as a possible conflict. Only the VMs with that MAC are read, and
the card's link-local address, which comes from the MAC, is left out.
The Administration menu gets "VM report (.xlsx)", a workbook for
superusers. The VMs sheet has one row per VM on every compute, shut-off
ones and those the panel has not registered too: name,
state, host, OS and OS family, vCPUs, memory, IP addresses and where they
came from, MAC addresses, owners, and whether the panel has registered
it. A Coverage sheet says when it was made and what could not be read.
The addresses come from the same live reading as the address search,
now shared by both. The OS is the guest agent's (libvirt's guest info)
or, without one, the libosinfo id a tool such as virt-install wrote into
the definition. Link-local IPv6 addresses are left out of the report.
Text cells stay text: a VM name starting with "=" is not a formula.
openpyxl 3.1.5 writes the workbook.
The Instances page's filters and the address search become one control:
the state list, the search box and a "Find IP" button. "Find IP" opens a
dialog (with the address typed in the search box, if any). An IP address
or a network (CIDR) finds the running VMs whose network cards use it,
read live from libvirt within 5 seconds: libvirt's DHCP leases, the host's
ARP table and the guest agent. Only addresses on the VM's own NICs count,
so interfaces inside the guest such as docker0 never match. Each match
shows the VM, its MAC, where the address came from, and a possible-
conflict mark when two VMs report one address. A coverage line says how
many running VMs were checked, how many had no connected guest agent (a
static address on them cannot be seen), which agents could not be read,
which hosts were not reached or not finished in time, and when it was
read.
The hosts are read in parallel: leases and ARP first, then each guest
agent on its own thread, so an agent that never answers holds back only
its own VM; libvirt waits for an agent without limit, so a search starts
only while its threads fit in 16 search threads in the worker.
Administrators search every compute, VMs the panel has not registered
yet included, and see the hosts; other users find only their own VMs and
see no hosts or host errors.
