SR-IOV Live Migration

Live migration refers to the process of moving a guest virtual machine (VM) running on one physical host to another host without disrupting normal operations or causing other adverse effects for the end user.

Using the Migration process is useful for:

• Load balancing

• Hardware independence

• Energy saving

• Geographic migration

• Fault tolerance

Migration works by sending the state of the guest virtual machine's memory and any virtualized devices to a destination host physical machine. Migrations can be performed live or not, in the live case, the migration will not disrupt the user operations and it will be transparent to it as explained in the following sections. 

Non-Live Migration

When using the non-live migration process, the Hypervisor suspends the guest virtual machine, then moves an image of the guest virtual machine's memory to the destination host physical machine. The guest virtual machine is then resumed on the destination host physical machine, and the memory the guest virtual machine used on the source host physical machine is freed. The time it takes to complete such a migration depends on the network bandwidth and latency. If the network is experiencing heavy use or low bandwidth, the migration will take longer then desired.

Live Migration

When using the Live Migration process, the guest virtual machine continues to run on the source host physical machine while its memory pages are transferred to the destination host physical machine. During migration, the Hypervisor monitors the source for any changes in the pages it has already transferred and begins to transfer these changes when all of the initial pages have been transferred.

It also estimates transfer speed during migration, so when the remaining amount of data to transfer will take a certain configurable period of time, it will suspend the original guest virtual machine, transfer the remaining data, and resume the same guest virtual machine on the destination host physical machine.

MLX5 VF Live Migration

The purpose of this section is to demonstrate how to perform basic live migration of a QEMU VM with an MLX5 VF assigned to it. This section does not explains how to create VMs either using libvirt or directly via QEMU.

Requirements

The below are the requirements for working with MLX5 VF Live Migration.

Setup

NVCONFIG

SR-IOV should be enabled and be configured to support the required number of VFs as of enabling live migration. This can be achieved by the below command:

mlxconfig -d *<PF_BDF>* s SRIOV_EN=1 NUM_OF_VFS=4 VF_MIGRATION_MODE=2

Where:

• SRIOV_EN – Enable SR-IOV.

• NUM_OF_VFS – The total number of VFs that can be supported, for each PF.

• VF_MIGRATION_MODE – Defines support for VF migration:

  • 0x0 – DEVICE_DEFAULT

  • 0x1 – MIGRATION_DISABLED

  • 0x2 – MIGRATION_ENABLED

Kernel Configuration

Needs to be compiled with driver MLX5_VFIO_PCI enabled. (i.e. CONFIG_MLX5_VFIO_PCI).

To load the driver, run:

modprobe mlx5_vfio_pci

QEMU

Needs to be compiled with VFIO_PCI enabled (this is enabled by default).

Host Preparation

As stated earlier, creating the VMs is beyond the scope of this guide and we assume that they are already created. However, the VM configuration should be a migratable configuration, similarly to how it is done without SR-IOV VFs.

Over libvirt

1. Set the PF in the "switchdev" mode (on both the source and destination hosts).

   devlink dev eswitch set pci/<PF_BDF> mode switchdev
   

2. Create the VFs that will be assigned to the VMs (on both the source and destination hosts).

   echo "1" > /sys/bus/pci/devices/<PF_BDF>/sriov_numvfs
   

3. Set the VFs as migration capable (on both the source and destination hosts).

   1. See the name and PCI path (VF_BDF) of the VFs, run:

      lspci -D | grep nox
      

   2. Unbind the VFs from mlx5_core, run: 

      echo '<VF_BDF>' > /sys/bus/pci/drivers/mlx5_core/unbind
      

   3. Determine the VF’s physical function (PF_BDF) using the VF_BDF from step 3.a.:

      ls -la /sys/bus/pci/devices/<VF_BDF>/physfn
      

   4. Use devlink to set each VF as migration capable, using the PF_BDF from step c, run:

      devlink port function set pci/<PF_BDF>/1 migratable enable
      

   5. Check migration enabled using command

      devlink port show pci/<PF_BDF>/1
      

4. Assign the VFs to the VMs (in both source and destination VMs).

   1. Get the desired VM names:

      virsh list --all
      

   2. Edit the VMs XML file, run: 

      virsh edit <VM_NAME>
      

   3. Assign the VFs to the VM by adding the following under the "devices" tag:

      XML

      <hostdev mode='subsystem' type='pci' managed='no'>
        <driver name='vfio'/>
        <source>
          <!-- below is the VF BDF as seen on the Host.
          Note: the VF BDF can be different on the source
          and destination Hosts -->
          <address domain='0x0000' bus='0x08' slot='0x00' function='0x2'/>
        </source>
        <!-- below is the VF BDF as it will appear on the VM.
        Note that the VF BDF must be the same on both the source
        and destination VMs -->
        <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </hostdev>
      

5. Set the destination VM in incoming mode.

   1. Edit the destination VM XML file, run: 

      virsh edit <VM_NAME>
      

   2. Set the destination VM in migration incoming mode by adding the following under "domain" tag: 

      <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
        [...]
        <qemu:commandline>
          <qemu:arg value='--incoming'/>
          <qemu:arg value='tcp:0:<DEST_PORT>'/>
        </qemu:commandline>
      </domain>
      

6. Bind the VFs to mlx5_vfio_pci driver (on both the source and destination hosts).

1. 1. Identify which VF_BDF to detach:

      virsh nodedev-list
      

   2. Detach the VFs from libvirt management, run: 

      virsh nodedev-detach pci_<VF_BDF>
      

   3. Check which driver was used:

      lspci -k | less
      

      Expected output:

      Kernel driver in use: mlx5_vfio_pci
      

   4. If you see that the kernel driver in use is vfio-pci, do the following:

      1. Rebind the VF to mlx5_vfio_pci:

         echo '<VF_BDF>' > /sys/bus/pci/drivers/vfio-pci/unbind
         

      2. Set driver override, run: 

         echo 'mlx5_vfio_pci' > /sys/bus/pci/devices/<VF_BDF>/driver_override
         

      3. Bind the VFs to mlx5_vfio_pci driver, run: 

         echo '<VF_BDF>' > /sys/bus/pci/drivers/mlx5_vfio_pci/bind
         

Directly over QEMU

1. Set the PF in "switchdev" mode. 

   devlink dev eswitch set pci/<PF_BDF> mode switchdev
   

2. Create the VFs that will be assigned to the VMs. 

   echo "1" > /sys/bus/pci/devices/<PF_BDF>/sriov_numvfs
   

3. Set the VFs as migration capable.

   1. See the name of the VFs, run: 

      devlink port show
      

   2. Unbind the VFs from mlx5_core, run: 

      echo '<VF_BDF>' > /sys/bus/pci/drivers/mlx5_core/unbind
      

   3. Use devlink to set each VF as migration capable, run: 

      devlink port function set pci/<PF_BDF>/1 migratable enable
      

4. Bind the VFs to mlx5_vfio_pci driver:

   1. Set driver override, run: 

      echo 'mlx5_vfio_pci' > /sys/bus/pci/devices/<VF_BDF>/driver_override
      

   2. Bind the VFs to mlx5_vfio_pci driver, run: 

      echo '<VF_BDF>' > /sys/bus/pci/drivers/mlx5_vfio_pci/bind
      

Running the Migration

Over libvirt

1. Check if the VM is running:

   virsh domstate <VM_NAME>
   

   If the output shows running, destroy the VM:

   virsh destroy <VM_NAME>
   

2. Start the VMs on the source and destination hosts: 

   virsh start <VM_NAME>
   

3. Enable switchover-ack QEMU migration capability. Run the following commands both in source and destination: 

   virsh qemu-monitor-command <VM_NAME> --hmp "migrate_set_capability return-path on"
   

   virsh qemu-monitor-command <VM_NAME> --hmp "migrate_set_capability switchover-ack on"
   

4. [Optional] Configure the migration bandwidth and downtime limit in source side: 

   virsh qemu-monitor-command <VM_NAME> --hmp "migrate_set_parameter max-bandwidth <VALUE>"
   virsh qemu-monitor-command <VM_NAME> --hmp "migrate_set_parameter downtime-limit <VALUE>"
   

5. Start migration by running the migration command in source side:

   virsh qemu-monitor-command <VM_NAME> --hmp "migrate -d tcp:<DEST_IP>:<DEST_PORT>"
   

6. Check the migration status by running the info command in source side:

   virsh qemu-monitor-command <VM_NAME> --hmp "info migrate"
   

Directly over QEMU

1. Start the VM in the source Host with the VF assigned to it:

   qemu-system-x86_64 [...] -device vfio-pci,host=<VF_BDF>,id=mlx5_1
   

2. Start the VM in the destination Host with the VF assigned to it and with the "incoming" parameter:

   qemu-system-x86_64 [...] -device vfio-pci,host=<VF_BDF>,id=mlx5_1 --incoming tcp:0:<DEST_PORT>
   

3. Enable switchover-ack QEMU migration capability. Run the following commands in QEMU monitor, both in source and destination:

   migrate_set_capability return-path on
   

   migrate_set_capability switchover-ack on
   

4. [Optional] Configure the migration bandwidth and downtime limit in source side:

   migrate_set_parameter max-bandwidth <VALUE>
   migrate_set_parameter downtime-limit <VALUE>
   

5. Start migration by running the migration command in QEMU monitor in source side:

   migrate -d tcp:<DEST_IP>:<DEST_PORT>
   

6. Check the migration status by running the info command in QEMU monitor in source side:

   info migrate
   

Migration with Multi-port vHCA

Enables the usage of a dual port virtual HCA (vHCA) to share RDMA resources (e.g., MR, CQ, SRQ, PDs) across the two Ethernet (RoCE) NIC network ports and display the NIC as a dual port device.

Multi-port vHCA (MPV) VF is made of 2 "regular" VFs, one VF of each port. Creating a migratable MPV VF requires the same steps as regular VF (see the steps in section “Over libvirt”). The steps should be performed on each of the NIC ports. MPV VFs traffic cannot be configured with OVS. TC rules must be defined to configure the MPV VFs traffic.

Notes

Last updated: July 08, 2026