j3d1/webvirtcloud
1
0
Fork 0
mirror of https://github.com/retspen/webvirtcloud synced 2026-11-09 03:36:48 +00:00
Code Issues Projects Releases Wiki Activity

fix(instances): a migration that does not finish in time is cancelled, not cut off

Browse source 
A VM that changes its memory faster than the network carries it never
finishes a live migration without auto converge: measured with a guest
rewriting about 1.2 GB/s over an 8.5 Gbit/s link. The migration then
ran until gunicorn ended the request after 600 s. libvirt cancelled it
when the request went away and the VM stayed on the source, but the
user got an error page and nothing was logged.

The app now cancels a migration that has not finished after 540 s
(MIGRATION_TIME_LIMIT, below the 600 s of gunicorn.conf.py; a test
keeps it below). The failure is reported and logged like any other:
the migration was cancelled, the VM stays on the source host, and a
VM like that needs auto converge. A timer that fires as the migration
ends finishes before the compute locks are released, and an error
the cancel did not cause keeps its own message.
This commit is contained in:
catborise 2026-10-06 16:56:22 +03:00
parent 1016f745c2
commit 043c24a208
4 changed files with 120 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

44
vrtManager/instance.py
View file

@ -4,6 +4,7 @@ import json
import logging
import os.path
import string
import threading
import time
logger = logging.getLogger(__name__)
@ -113,7 +114,7 @@ class wvmInstances(wvmConnect):
dom = self.get_instance(name)
dom.resume()
def moveto(self, conn, name, live, unsafe, offline, autoconverge=False, compress=False, uri=None):
def moveto(self, conn, name, live, unsafe, offline, autoconverge=False, compress=False, uri=None, timeout=None):
"""Migrates the VM of conn (a wvmInstance on the source) here.
The mode follows the VM's state, read once: a running VM migrates
@ -122,8 +123,9 @@ class wvmInstances(wvmConnect):
fit the state is refused. unsafe, autoconverge (live only) and
compress apply to a running VM. uri: the native migration URI
(tcp://address) on the destination; None lets libvirt use the
destination's own hostname. Returns the mode and options used, for
the log."""
destination's own hostname. timeout: seconds after which a migration
that has not finished is cancelled (None: no limit). Returns the mode
and options used, for the log."""
state = conn.get_status()
if state == 5:
if live:
@ -179,9 +181,43 @@ class wvmInstances(wvmConnect):
# dom.migrate.
self.move_definition(dom)
else:
dom.migrate(self.wvm, flags, None, uri, 0)
self.migrate_within(dom, flags, uri, timeout)
return ", ".join(mode)
def migrate_within(self, dom, flags, uri, timeout):
"""dom.migrate, cancelled when it has not finished after timeout
seconds: a VM that changes its memory faster than the network carries
it never finishes, and the request would be ended by the web server
with no word. Cancelled, the VM stays on the source."""
cancelled = threading.Event()
def cancel():
with contextlib.suppress(libvirtError): # finished meanwhile
dom.abortJob()
cancelled.set()
timer = threading.Timer(timeout, cancel) if timeout else None
if timer:
timer.daemon = True
timer.start()
try:
dom.migrate(self.wvm, flags, None, uri, 0)
except libvirtError as error:
if timer:
timer.cancel()
timer.join() # a cancel under way ends before the locks go
if cancelled.is_set():
raise util.OperationError(
f"The migration did not finish within {timeout} s and was cancelled ({error}); the VM stays on "
"the source host. A VM that changes its memory faster than the network carries it needs auto "
"converge."
) from error
raise
finally:
if timer:
timer.cancel()
timer.join()
def missing_disks(self, definitions):
"""(target, source) of the disks in the VM definitions (XML) that are
not volumes of this host's active storage pools: the VM's own files,