A VM that changes its memory faster than the network carries it never
finishes a live migration without auto converge: measured with a guest
rewriting about 1.2 GB/s over an 8.5 Gbit/s link. The migration then
ran until gunicorn ended the request after 600 s. libvirt cancelled it
when the request went away and the VM stayed on the source, but the
user got an error page and nothing was logged.
The app now cancels a migration that has not finished after 540 s
(MIGRATION_TIME_LIMIT, below the 600 s of gunicorn.conf.py; a test
keeps it below). The failure is reported and logged like any other:
the migration was cancelled, the VM stays on the source host, and a
VM like that needs auto converge. A timer that fires as the migration
ends finishes before the compute locks are released, and an error
the cancel did not cause keeps its own message.
What migrate_instance did around the migration itself:
- It checked the destination with a 1 s TCP probe of the libvirt port,
which passes while libvirt is down or refuses the login. It now uses
the compute's connection status, as the rest of the app does.
- Two requests for the same VM waited for the compute locks one after
the other; the second then migrated with the VM's old location. The
VM's record is read again under the locks, and the request is
refused when the VM is no longer on the source.
- Restoring autostart on the destination came after the migration but
before the database move; when it failed, the VM ran on the
destination while the app recorded it on the source. Autostart is
now read from libvirt and a failure to restore it is only logged.
- A database failure after a successful migration was logged as
nothing; the log now says the VM migrated and recording it failed.
- When the destination's list had already made a record for the VM,
the merge kept the owners but dropped the template mark.
