DOCA Virtio-net Service Guide

This guide provides instructions on how to use the DOCA virtio-net service container on top of NVIDIA® BlueField®-3 networking platform.

Introduction

NVIDIA® BlueField® virtio-net enables users to create virtio-net PCIe devices in the system where the BlueField is connected. In a traditional virtualization environment, virtio-net devices can be emulated by QEMU from the hypervisor, or offloading part of the work (e.g., dataplane) to the NIC (e.g., vDPA). Compared to those solutions, virtio-net PCIe devices offload both data and control plane to the BlueField networking device. The PCIe virtio-net devices exposed to the hypervisor do not depend on QEMU or other software emulators/vendor drivers from the guest OS.

The solution is based on BlueField family technology on top of virtual switch and OVS, so that virtio-net devices can benefit from the full SDN and hardware offload methodologies.

Virtio-net Controller SystemD Service

Virtio-net-controller is a systemd service which runs the BlueField with a command-line interface (CLI) frontend to communicate with the service running in the background. The controller systemd service is enabled by default and runs automatically after certain firmware configurations are deployed. 

The processes virtio_net_emu and virtio_net_ha are created to manage live update and high availability.

Virtio-net Deployment

Updating OS Image on BlueField

To install the BFB bundle on the NVIDIA® BlueField®, run the following command from the Linux hypervisor:

[host]# sudo bfb-install --rshim <rshimN> --bfb <image_path.bfb>

For more information, refer to section "Deploying BlueField Software Using BFB from Host" in the NVIDIA BlueField DPU BSP documentation.

Updating NIC Firmware

From the BlueField networking platform, run:

[dpu]# sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update

For more information, refer to section "Upgrading Firmware" in the NVIDIA DOCA Installation Guide for Linux.

Configuring NIC Firmware

As default, DPU should be configured in DPU mode. A simple way to confirm DPU is running at DPU mode is to log into the BlueField Arm system and check if p0 and pf0hpf both exists by running command below.

[dpu]# ip link show

Virtio-net full emulation only works in DPU mode. For more information about DPU mode configuration, please refer to BlueField Modes of Operation.

Before enabling the virtio-net service, configure firmware via mlxconfig tool is required. There are examples on typical configurations, the table listed relevant mlxconfig entry descriptions.

The maximum number of supported devices is listed below. It does not apply when there are hot-plug and VF created at the same time.

Static PF

Static PF is defined as virtio-net PFs which are persistent even after DPU or host power cycle. It also supports creating SR-IOV VFs.

The following is an example for enabling the system with 4 static PFs (VIRTIO_NET_EMULATION_NUM_PF) only:

[dpu]# mlxconfig -d 03:00.0 s \
VIRTIO_NET_EMULATION_ENABLE=1 \
VIRTIO_NET_EMULATION_NUM_PF=4 \
VIRTIO_NET_EMULATION_NUM_VF=0 \
VIRTIO_NET_EMULATION_NUM_MSIX=64 \
PCI_SWITCH_EMULATION_ENABLE=0 \
PCI_SWITCH_EMULATION_NUM_PORT=0 \
PER_PF_NUM_SF=1 \
PF_TOTAL_SF=64 \
PF_BAR2_ENABLE=0 \
PF_SF_BAR_SIZE=8 \
SRIOV_EN=0

Hotplug PF

Hotplug PF is defined as virtio-net PFs which can be hotplugged or unplugged dynamically after the system comes up. 

The following is an example for enabling 16 hotplug PFs (PCI_SWITCH_EMULATION_NUM_PORT):

[dpu]# mlxconfig -d 03:00.0 s \
VIRTIO_NET_EMULATION_ENABLE=1 \
VIRTIO_NET_EMULATION_NUM_PF=0 \
VIRTIO_NET_EMULATION_NUM_VF=0 \
VIRTIO_NET_EMULATION_NUM_MSIX=64 \
PCI_SWITCH_EMULATION_ENABLE=1 \
PCI_SWITCH_EMULATION_NUM_PORT=16 \
PER_PF_NUM_SF=1 \
PF_TOTAL_SF=64 \
PF_BAR2_ENABLE=0 \
PF_SF_BAR_SIZE=8 \
SRIOV_EN=0 

SR-IOV VF

SR-IOV VF is defined as virtio-net VFs created on top of PFs. Each VF gets an individual virtio-net PCIe devices.

The following is an example for enabling 126 VFs per static PF—504 (4 PF x 126) VFs in total:

[dpu]# mlxconfig -d 03:00.0 s \
VIRTIO_NET_EMULATION_ENABLE=1 \
VIRTIO_NET_EMULATION_NUM_PF=4 \
VIRTIO_NET_EMULATION_NUM_VF=126 \
VIRTIO_NET_EMULATION_NUM_MSIX=64 \
VIRTIO_NET_EMULATION_NUM_VF_MSIX=6 \
PCI_SWITCH_EMULATION_ENABLE=0 \
PCI_SWITCH_EMULATION_NUM_PORT=0 \
PER_PF_NUM_SF=1 \
PF_TOTAL_SF=512 \
PF_BAR2_ENABLE=0 \
PF_SF_BAR_SIZE=8 \
NUM_VF_MSIX=0 \
SRIOV_EN=1

PF/VF Combinations

Creating static/hotplug PFs and VFs at the same time is supported.

The total sum of PCIe functions to the external host must not exceed 1008. For example:

• If there are 2 PFs with no VFs (NUM_OF_VFS=0) and there is 1 RShim, then the remaining static functions is 1005 (1008-3).

• If 1 virtio-net PF is configured (VIRTIO_NET_EMULATION_NUM_PF=1), then up to 1004 virtio-net VFs can be configured (VIRTIO_NET_EMULATION_NUM_VF=1004)

• If 2 virtio-net PF (VIRTIO_NET_EMULATION_NUM_PF=2), then up to 502 virtio-net VFs can be configured (VIRTIO_NET_EMULATION_NUM_VF=502)

The following is an example for enabling 15 hotplug PFs, 2 static PFs, and 200 VFs (2 PFs x 100):

[dpu]# mlxconfig -d 03:00.0 s \
VIRTIO_NET_EMULATION_ENABLE=1 \
VIRTIO_NET_EMULATION_NUM_PF=2 \
VIRTIO_NET_EMULATION_NUM_VF=100 \
VIRTIO_NET_EMULATION_NUM_MSIX=10 \
VIRTIO_NET_EMULATION_NUM_VF_MSIX=6 \
PCI_SWITCH_EMULATION_ENABLE=1 \
PCI_SWITCH_EMULATION_NUM_PORT=15 \
PER_PF_NUM_SF=1 \
PF_TOTAL_SF=256 \
PF_BAR2_ENABLE=0 \
PF_SF_BAR_SIZE=8 \
NUM_VF_MSIX=0 \
SRIOV_EN=1

System Configuration

Host System Configuration

For hotplug device configuration, it is recommended to modify the hypervisor OS kernel boot parameters and add the options below:

pci=realloc

For SR-IOV configuration, first enable SR-IOV from the host. 

Make sure to add the following options to Linux boot parameter.

intel_iommu=on iommu=pt

pci 0000:84:00.0: [1af4:1041] type 7f class 0xffffff
pci 0000:84:00.0: unknown header type 7f, ignoring device

BlueField System Configuration

Virtio-net full emulation is based on ASAP^2. For each virtio-net device created from host side, there is an SF representor created to represent the device from the BlueField side. It is necessary to have the SF representor in the same OVS bridge of the uplink representor.

The SF representor name is designed in a fixed pattern to map different type of devices.

For example, the first static PF gets the SF representor of en3f0pf0sf1000 and the second hotplug PF gets the SF representor of en3f0pf0sf2001. It is recommended to verify the name of the SF representor from the sf_rep_net_device field in the output of virtnet list.

[dpu]# virtnet list
{
  ...
  "devices": [
    {
      "pf_id": 0,
      "function_type": "static PF",
      "transitional": 0,
      "vuid": "MT2151X03152VNETS0D0F2",
      "pci_bdf": "14:00.2",
      "pci_vhca_id": "0x2",
      "pci_max_vfs": "0",
      "enabled_vfs": "0",
      "msix_num_pool_size": 0,
      "min_msix_num": 0,
      "max_msix_num": 32,
      "min_num_of_qp": 0,
      "max_num_of_qp": 15,
      "qp_pool_size": 0,
      "num_msix": "64",
      "num_queues": "8",
      "enabled_queues": "7",
      "max_queue_size": "256",
      "msix_config_vector": "0x0",
      "mac": "D6:67:E7:09:47:D5",
      "link_status": "1",
      "max_queue_pairs": "3",
      "mtu": "1500",
      "speed": "25000",
      "rss_max_key_size": "0",
      "supported_hash_types": "0x0",
      "ctrl_mac": "D6:67:E7:09:47:D5",
      "ctrl_mq": "3",
      "sf_num": 1000,
      "sf_parent_device": "mlx5_0",
      "sf_parent_device_pci_addr": "0000:03:00.0",
      "sf_rep_net_device": "en3f0pf0sf1000",
      "sf_rep_net_ifindex": 15,
      "sf_rdma_device": "mlx5_4",
      "sf_cross_mkey": "0x18A42",
      "sf_vhca_id": "0x8C",
      "sf_rqt_num": "0x0",
      "aarfs": "disabled",
      "dim": "disabled"
    }
  ]
 }

Once SF representor name is located, add it to the same OVS bridge of the corresponding uplink representor and make sure the SF representor is up:

[dpu]# ovs-vsctl show
f2c431e5-f8df-4f37-95ce-aa0c7da738e0
    Bridge ovsbr1
        Port ovsbr1
            Interface ovsbr1
                type: internal
        Port en3f0pf0sf0
            Interface en3f0pf0sf0
        Port p0
            Interface p0 
[dpu]# ovs-vsctl add-port ovsbr1 en3f0pf0sf1000
[dpu]# ovs-vsctl show
f2c431e5-f8df-4f37-95ce-aa0c7da738e0
    Bridge ovsbr1
        Port ovsbr1
            Interface ovsbr1
                type: internal
        Port en3f0pf0sf0
            Interface en3f0pf0sf0
        Port en3f0pf0sf1000
            Interface en3f0pf0sf1000
        Port p0
            Interface p0
[dpu]# ip link set dev en3f0pf0sf1000 up

Usage

After firmware/system configuration and after system power cycle, the virtio-net devices should be ready to deploy.

First, make sure that mlxconfig options take effect correctly by issuing the following command: 

[dpu]# mlxconfig -d 03:00.0 -e q | grep -i \*
*        PER_PF_NUM_SF                               False(0)        True(1)         True(1)
*        NUM_OF_VFS                                  16              0               0
*        PF_BAR2_ENABLE                              True(1)         False(0)        False(0)
*        PCI_SWITCH_EMULATION_NUM_PORT               0               8               8
*        PCI_SWITCH_EMULATION_ENABLE                 False(0)        True(1)         True(1)
*        VIRTIO_NET_EMULATION_ENABLE                 False(0)        True(1)         True(1)
*        VIRTIO_NET_EMULATION_NUM_VF                 0               126             126
*        VIRTIO_NET_EMULATION_NUM_PF                 0               1               1
*        VIRTIO_NET_EMULATION_NUM_MSIX               2               64              64
*        VIRTIO_NET_EMULATION_NUM_VF_MSIX            0               64              64
*        PF_TOTAL_SF                                 0               508             508
*        PF_SF_BAR_SIZE                              0               8               8

If the system is configured correctly, virtio-net-controller service should be up and running. If the service does not appear as active, double check the firmware/system configurations above.

[dpu]# systemctl status virtio-net-controller.service
● virtio-net-controller.service - Nvidia VirtIO Net Controller Daemon
   Loaded: loaded (/etc/systemd/system/virtio-net-controller.service; enabled; vendor preset: disabled)
   Active: active (running)
     Docs: file:/opt/mellanox/mlnx_virtnet/README.md
 Main PID: 30715 (virtio_net_cont)
    Tasks: 55
   Memory: 11.7M
   CGroup: /system.slice/virtio-net-controller.service
           ├─30715 /usr/sbin/virtio_net_controller
           ├─30859 virtio_net_emu
           └─30860 virtio_net_ha 

To reload or restart the service, run:

[dpu]# systemctl restart virtio-net-controller.service

Hotplug PF Devices

Creating PF Devices

1. To create a hotplug virtio-net device, run:

   [dpu]# virtnet hotplug -i mlx5_0 -f 0x0 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024
   
   

   
   

   Refer to "Virtnet CLI Commands" for full usage.

   This command creates one hotplug virtio-net device with MAC address 0C:C4:7A:FF:22:93, MTU 1500, and 3 virtio queues with a depth of 1024 entries. The device is created on the physical port of mlx5_0. The device is uniquely identified by its index. This index is used to query and update device attributes. If the device is created successfully, an output similar to the following appears:

   {
     "bdf": "15:00.0",
     "vuid": "MT2151X03152VNETS1D0F0",
     "id": 0,
     "transitional": 0,
     "sf_rep_net_device": "en3f0pf0sf2000",
     "mac": "0C:C4:7A:FF:22:93",
     "errno": 0,
     "errstr": "Success"
   }
   
   

2. Add the representor port of the device to the OVS bridge and bring it up. Run:

   [dpu]# ovs-vsctl add-port <bridge> en3f0pf0sf2000
   [dpu]# ip link set dev en3f0pf0sf2000 up
   
   

   Once steps 1-2 are completed, the virtio-net PCIe device should be available from hypervisor OS with the same PCIe BDF.

   [host]# lspci | grep -i virtio
   15:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
   
   

3. Probe virtio-net driver (e.g., kernel driver): 

   [host]# modprobe -v virtio-pci && modprobe -v virtio-net
   
   

   
   

4. The virtio-net device should be created. There are two ways to locate the net device:

   • Check the dmesg from the host side for the corresponding PCIe BDF:

     [host]# dmesg | tail -20 | grep 15:00.0 -A 10 | grep virtio_net
     [3908051.494493] virtio_net virtio2 ens2f0: renamed from eth0
     
     

     
     

   • Check all net devices and find the corresponding MAC address:

     [host]# ip link show | grep -i "0c:c4:7a:ff:22:93" -B 1
     31: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
         link/ether 0c:c4:7a:ff:22:93 brd ff:ff:ff:ff:ff:ff
     
     

     
     

5. Check that the probed driver and its BDF match the output of the hotplug device:

   [host]# ethtool -i ens2f0
   driver: virtio_net
   version: 1.0.0
   firmware-version:
   expansion-rom-version:
   bus-info: 0000:15:00.0
   supports-statistics: yes
   supports-test: no
   supports-eeprom-access: no
   supports-register-dump: no
   supports-priv-flags: no
   
   

   
   

Now the hotplug virtio-net device is ready to use as a common network device.

Destroying PF Devices

To hot-unplug a virtio-net device, run:

[dpu]# virtnet unplug -p 0
{'id': '0x1'}
{
  "errno": 0,
  "errstr": "Success"
}

The hotplug device and its representor are destroyed.

Host-Aware Attention Button Mode (AB Mode)

The Host-Aware Attention Button (AB) mode provides a coordinated mechanism for device removal utilizing the PCIe Attention Button. This mode ensures the host operating system has the opportunity to gracefully shut down the device driver prior to physical or logical removal.

When utilizing AB mode (-w 3), the hotplug and unplug lifecycle operates as follows:

1. Hotplug initiation: The device is created with AB mode awareness, a state that persists for the lifetime of the device.

2. Unplug notification: Upon an unplug request, the controller sends an Attention Button notification to the host, signaling a pending device removal.

3. Host acknowledgment: The host OS receives the notification and gracefully shuts down the associated driver.

4. Clean removal: Once the host acknowledges the shutdown, the device is cleanly removed from the system.

Usage Commands

Creating a device with AB mode:

[dpu]# virtnet hotplug -i mlx5_0 -f 0x0 -m 0C:C4:7A:FF:22:93 -t 1500 -n 3 -s 1024 -w 3

Removing a device with AB mode:

[dpu]# virtnet unplug -p 0 -w 3

Verification and Querying

Checking AB Mode Support

Run the virtnet list command to verify if the underlying controller supports AB mode. Look for the hp_host_aware_ab_supported key in the JSON output.

{
    "controller": {
        "hp_host_awareness_supported": "1",
        "hp_host_aware_ab_supported": "1"
    }
}

Querying Device HP Mode

To verify which hotplug mode a specific device was created with, use the virtnet query command. An awareness mode of 3 indicates AB mode.

{
    "hp_host_awareness_mode": 3
}

Default AB Mode Configuration

To automate AB mode for all hotplug and unplug operations without manually specifying the -w 3 flag on the CLI, configure the virtnet.conf file:

force_ab_hotplug_default=1

Behavior when enabled:

• Hotplug operations automatically default to AB mode (-w 3).

• Unplug operations automatically default to AB mode (-w 3).

• Explicit -w CLI options will always override this configuration.

• If the underlying Host Channel Adapter (HCA) does not support AB mode, the controller will safely fall back to mode 0.

SR-IOV VF Devices

Creating SR-IOV VF Devices

After configuring the firmware and BlueField/host system with correct configuration, users can create SR-IOV VFs.

The following procedure provides an example of creating one VF on top of one static PF:

1. Locate the virtio-net PFs exposed to the host side:

   [host]# lspci | grep -i virtio
   14:00.2 Network controller: Red Hat, Inc. Virtio network device
   
   

2. Verify that the PCIe BDF matches the backend device from the BlueField side:

   [dpu]# virtnet list
   {
     ...
      "devices": [
       {
         "pf_id": 0,
         "function_type": "static PF",
         "transitional": 0,
         "vuid": "MT2151X03152VNETS0D0F2",
         "pci_bdf": "14:00.2",
         "pci_vhca_id": "0x2",
         "pci_max_vfs": "0",
         "enabled_vfs": "0",
         "msix_num_pool_size": 0,
         "min_msix_num": 0,
         "max_msix_num": 32,
         "min_num_of_qp": 0,
         "max_num_of_qp": 15,
         "qp_pool_size": 0,
         "num_msix": "64",
         "num_queues": "8",
         "enabled_queues": "7",
         "max_queue_size": "256",
         "msix_config_vector": "0x0",
         "mac": "D6:67:E7:09:47:D5",
         "link_status": "1",
         "max_queue_pairs": "3",
         "mtu": "1500",
         "speed": "25000",
         "rss_max_key_size": "0",
         "supported_hash_types": "0x0",
         "ctrl_mac": "D6:67:E7:09:47:D5",
         "ctrl_mq": "3",
         "sf_num": 1000,
         "sf_parent_device": "mlx5_0",
         "sf_parent_device_pci_addr": "0000:03:00.0",
         "sf_rep_net_device": "en3f0pf0sf1000",
         "sf_rep_net_ifindex": 15,
         "sf_rdma_device": "mlx5_4",
         "sf_cross_mkey": "0x18A42",
         "sf_vhca_id": "0x8C",
         "sf_rqt_num": "0x0",
         "aarfs": "disabled",
         "dim": "disabled"
       }
     ]
    }
   
   

3. Probe virtio_pci and virtio_net modules from the host:

   [host]# modprobe -v virtio-pci && modprobe -v virtio-net 
   
   

   The PF net device should be created.

   [host]# ip link show | grep -i "4A:82:E3:2E:96:AB" -B 1
   21: ens2f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
       link/ether 4a:82:e3:2e:96:ab brd ff:ff:ff:ff:ff:ff
   
   

   The MAC address and PCIe BDF should match between the BlueField side (virtnet list) and host side (ethtool).

   [host]# ethtool -i ens2f2
   driver: virtio_net
   version: 1.0.0
   firmware-version:
   expansion-rom-version:
   bus-info: 0000:14:00.2
   supports-statistics: yes
   supports-test: no
   supports-eeprom-access: no
   supports-register-dump: no
   supports-priv-flags: no
   
   

4. To create SR-IOV VF devices on the host, run the following command with the PF PCIe BDF (0000:14:00.2 in this example):

   [host]# echo 1 > /sys/bus/pci/drivers/virtio-pci/0000\:14\:00.2/sriov_numvfs
   
   

   1 extra virtio-net device is created from the host: 

   [host]# lspci | grep -i virtio
   14:00.2 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
   14:00.4 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
   
   

   The BlueField side shows the VF information from virtnet list as well:

   [dpu]# virtnet list
       ...
       {
         "vf_id": 0,
         "parent_pf_id": 0,
         "function_type": "VF",
         "transitional": 0,
         "vuid": "MT2151X03152VNETS0D0F2VF1",
         "pci_bdf": "14:00.4",
         "pci_vhca_id": "0xD",
         "pci_max_vfs": "0",
         "enabled_vfs": "0",
         "num_msix": "12",
         "num_queues": "8",
         "enabled_queues": "7",
         "max_queue_size": "256",
         "msix_config_vector": "0x0",
         "mac": "16:FF:A2:6E:6D:A9",
         "link_status": "1",
         "max_queue_pairs": "3",
         "mtu": "1500",
         "speed": "25000",
         "rss_max_key_size": "0",
         "supported_hash_types": "0x0",
         "ctrl_mac": "16:FF:A2:6E:6D:A9",
         "ctrl_mq": "3",
         "sf_num": 3000,
         "sf_parent_device": "mlx5_0",
         "sf_parent_device_pci_addr": "0000:03:00.0",
         "sf_rep_net_device": "en3f0pf0sf3000",
         "sf_rep_net_ifindex": 18,
         "sf_rdma_device": "mlx5_5",
         "sf_cross_mkey": "0x58A42",
         "sf_vhca_id": "0x8D",
         "sf_rqt_num": "0x0",
         "aarfs": "disabled",
         "dim": "disabled"
        }
   
   

5. Add the corresponding SF representor to the OVS bridge as the virtio-net PF and bring it up. Run:

   [dpu]# ovs-vsctl add-port <bridge> en3f0pf0sf3000
   [dpu]# ip link set dev en3f0pf0sf3000 up
   
   

Now the VF is functional. 

[host]# echo 0 > /sys/bus/pci/drivers/virtio-pci/<virtio_pf_bdf>/sriov_drivers_autoprobe
[host]# echo <num_vfs> > /sys/bus/pci/drivers/virtio-pci/<virtio_pf_bdf>/sriov_numvfs

[host]# echo <virtio_vf_bdf> > /sys/bus/pci/drivers/virtio-pci/bind

[host]# echo 1 > /sys/bus/pci/drivers/virtio-pci/<virtio_pf_bdf>/sriov_drivers_autoprobe

Destroying SR-IOV VF Devices

To destroy SR-IOV VF devices on the host, run:

[host]# echo 0 > /sys/bus/pci/drivers/virtio-pci/<virtio_pf_bdf>/sriov_numvfs

Assigning Virtio-net Device to VM

All virtio-net devices (static/hotplug PF and VF) support PCIe passthrough to a VM. PCIe passthrough allows the device to get better performance in the VM.

Assigning a virtio-net device to a VM can be done via virt-manager or virsh command.

Locating Virtio-net Devices

All virtio-net devices can be scanned by the PCIe subsystem in hypervisor OS and displayed as a standard PCIe device. Run the following command to locate the virtio-net devices devices with its PCIe BDF.

[host]# lspci | grep 'Virtio network'
00:09.1 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)

Using virt-manager

Start virt-manager, run the following command:

[host]# virt-manager

Double-click the virtual machine and open its Properties. Navigate to Details → Add hardware → PCIe host device.

Choose a virtio-net device virtual function according to its PCIe device (e.g., 00:09.1), reboot or start the VM.

Using virsh Command

1. Run the following command to get the VM list and select the target VM by Name field:

   [host]# virsh list --all
    Id   Name                           State
   ----------------------------------------------
    1    host-101-CentOS-8.5           running
   
   

2. Edit the VMs XML file, run: 

   [host]# virsh edit <VM_NAME>
   
   

3. Assign the target virtio-net device PCIe BDF to the VM, using vfio as driver, replace BUS/SLOT/FUNCTION/BUS_IN_VM/SLOT_IN_VM/FUNCTION_IN_VM with corresponding settings.

   XML

   <hostdev mode='subsystem' type='pci' managed='no'>
     <driver name='vfio'/>
       <source>
         <address domain='0x0000' bus='<#BUS>' slot='<#SLOT>' function='<#FUNCTION>'/>
       </source>
     <address type='pci' domain='0x0000' bus='<#BUS_IN_VM>' slot='<#SLOT_IN_VM>' function='<#FUNCTION_IN_VM>'/>
   </hostdev>
   
   

   For example, assign target device 00.09.1 to the VM and its PCIe BDF within the VM is 01:00.0: 

   <hostdev mode='subsystem' type='pci' managed='no'>
     <driver name='vfio'/>
       <source>
         <address domain='0x0000' bus='0x00' slot='0x09' function='0x1'/>
       </source>
     <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
   </hostdev>
   
   

4. Destroy the VM if it is already started:

   [host]# virsh destory <VM_NAME>
   
   

5. Start the VM with new XML configuration:

   [host]# virsh start <VM_NAME>
   
   

Configuration File

Configuration File Options

The controller service has an optional JSON format configuration file which allows users to customize several parameters. The configuration file should be defined on the DPU at /opt/mellanox/mlnx_virtnet/virtnet.conf. This file is read every time the controller starts. 

---
version: 25.04
---
DPA_APPS:
  virtio-net:
    - partition: ROOT
      affinity_core:
        - core: 1
          num_EUs: 16
        - core: 3
          num_EUs: 16
        - core: 5
          num_EUs: 16

Configuration File Examples

Configuring LAG on Dual Port BlueField

Refer to "Link Aggregation" documentation for information on configuring BlueField in LAG mode.

Refer to the "Link Aggregation" page for information on configuring virtio-net in LAG mode.

Configuring Static PF on Dual Port BlueField

The following configures all static PFs to use mlx5_0 (port 0) as the data path device in a non-LAG configuration, and the default MAC and features for the PF:

{
  "ib_dev_p0": "mlx5_0",
  "ib_dev_p1": "mlx5_1",
  "ib_dev_for_static_pf": "mlx5_0",
  "is_lag": 0,
  "static_pf": {
    "mac_base": "08:11:22:33:44:55",
    "features": "0x230047082b"
  }
}

Configuring VF Specific Options

The following configures VFs with default parameters. With this configuration, each PF assigns the MAC based on mac_base up to 126 VFs. Each VF creates 4 queue pairs, with each queue having a depth of 256.

{
  "vf": {
    "mac_base": "06:11:22:33:44:55",
    "features": "0x230047082b",
    "vfs_per_pf": 126,
    "max_queue_pairs": 4,
    "max_queue_size": 256
  }
}

Virtio Live Migration Settings

The following table provides an example of configurations for the new options introduced for VirtIO Live Migration:

DPA Configuration (SPRD)

The Single Point of Resource Distribution (SPRD) is a centralized orchestration system that manages Data Path Accelerator (DPA) Execution Unit (EU) allocation across applications and Virtual HCAs (VHCAs). By centralizing this process, SPRD prevents ad-hoc per-application sizing and eliminates resource over-commitment.

Administrators define EU assignments to partitions and applications within a single YAML configuration file. The SPRD system validates this configuration, programs the hardware partitions and EU groups accordingly, and generates per-partition configuration files for the applications to consume.

Control and Integration

• CLI utility: The system is managed using the dpa-resource-mgmt command-line tool.

• Virtio-net integration: The virtio-net controller consumes the SPRD-generated output file via the dpa_partition configuration option to inherit its managed EU affinity.

SPRD Configuration Workflow for Virtio-net

Inspect Available Resources

Query the DPU to verify the ROOT partition and determine the set of free EUs before assigning them to virtio-net. 

dpa-resource-mgmt query -t resources -d mlx5_0

Create an Input SPRD YAML

Create an input file (e.g., input_vnet.yaml) that explicitly maps the virtio-net application to the ROOT partition and defines the required EUs per core.

Example for a virtio-net "solo app" deployment:

version: 26.01
DPA_APPS:
   "virtio-net":
       - partition: ROOT
         affinity_core:
             - core: 0
               num_EUs: 16
             - core: 1
               num_EUs: 16
             - core: 2
               num_EUs: 16
             - core: 3
               num_EUs: 16
             - core: 4
               num_EUs: 16
             - core: 5
               num_EUs: 16
             - core: 6
               num_EUs: 16
             - core: 7
               num_EUs: 16
             - core: 8
               num_EUs: 16
             - core: 9
               num_EUs: 16

Generate the SPRD Output YAML

Run the configuration command to generate the output file.

dpa-resource-mgmt config -d mlx5_0 -f input_vnet.yaml -v

Example SPRD Output & Validation

The generation command produces a parsed YAML file utilized by the application. The affinity_EUs list contains the absolute EU IDs reserved for virtio-net. During execution, SPRD converts these absolute IDs into partition-relative indices.

Example ROOT.yaml Output:

version: 26.01
DPA_APPS:
-   name: virtio-net
    number_of_affinity_EUs: 160
    affinity_EUs: [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 15, 16, 17, 19, 21, 22,
        23, 24, 26, 27, 28, 29, 30, 31, 33, 34, 36, 37, 39, 40, 41, 42, 43, 44, 45,
        48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 69,
        70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 89, 90,
        92, 93, 94, 95, 96, 97, 98, 99, 101, 102, 103, 104, 105, 106, 107, 108, 109,
        110, 111, 112, 113, 114, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125,
        126, 127, 129, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 143,
        144, 145, 147, 148, 149, 150, 151, 152, 154, 155, 156, 157, 158, 159, 160,
        161, 162, 163, 165, 166, 167, 168, 169, 172, 173, 174, 175, 177, 178, 179,
        181, 182, 183, 184, 185, 187, 188, 189]

Performance Validation (BlueField-3 DPU)

This specific SPRD configuration was validated in a "solo app" scenario (where virtio-net was the exclusive DPA application assigned EUs) to ensure all data-plane threads ran optimally across the 160 managed EUs.

Hardware and test profile:

• Devices: 32 full emulation virtio-net VFs (16 TX, 16 RX)

• Queue Configuration: Virtqueue (VQ) depth of 1024; 31 Queue Pairs (QPs) per device.

• CQE Moderation: Count = 32, Period = 32

• Traffic Profile: testpmd UDP, 16 streams, 64-byte message size.

Measured aggregate performance:

• RX-Only: 72.6 Mpps

• TX-Only: 81.05 Mpps

• Bidirectional (RX + TX): 100.36 Mpps total

Virtnet CLI Commands

User Front End CLI

To communicate with the virtio-net-controller backend service, a user frontend program, virtnet, is installed on the BlueField which is based on remote procedure call (RPC) protocol with JSON format output.

Hotplug

This command hotplugs a virtio-net PCIe PF device exposed to the host side.

Syntax

virtnet hotplug -i IB_DEVICE -m MAC -t MTU -n MAX_QUEUES -s MAX_QUEUE_SIZE [-h] [-u SF_NUM] [-f FEATURES] [-l] [-w HP_HOST_AWARENESS]

#virtnet list

{
        "controller":   {
                …
                "emulation_manager":    "mlx5_0",
                "max_hotplug_devices":  "15",
                "hp_host_awareness_supported":  "1",        // <===== new indication
                "max_virt_net_devices": "15",
                "max_virt_queues":      "256",
                ...

Output

Example

The following example of hot plugging one device with MAC address 0C:C4:7A:FF:22:93, MTU 1500, and 1 pair of virtual queue (QP) pair with a depth of 1024 entries. The device is created on the physical port of mlx5_0.

# virtnet hotplug -i mlx5_0 -m 0C:C4:7A:FF:22:93 -t 1500 -qp 1 -s 1024
{
  "bdf": "15:00.0",
  "vuid": "MT2151X03152VNETS1D0F0",
  "id": 0,
  "transitional": 0,
  "sf_rep_net_device": "en3f0pf0sf2000",
  "mac": "0C:C4:7A:FF:22:93",
  "errno": 0,
  "errstr": "Success"
}

Unplug

This command unplugs a virtio-net PCIe PF device.

Syntax

virtnet unplug [-h] [-p PF | -u VUID] [-w HP_HOST_AWARENESS] [-T HOTPLUG_POWER_OFF_TIMEOUT]

Only one of --pf  and --vuid is needed to unplug the device.

#virtnet list

{
        "controller":   {
                …
                "emulation_manager":    "mlx5_0",
                "max_hotplug_devices":  "15",
                "hp_host_awareness_supported":  "1",        // <===== new indication
                "max_virt_net_devices": "15",
                "max_virt_queues":      "256",
                ...

Output

Example

Unplug-hotplug device using the PF ID:

# virtnet unplug -p 0
{'id': '0x1'}
{
  "errno": 0,
  "errstr": "Success"
}

List

This command lists all existing virtio-net devices, with global information and individual information for each device.

Syntax

virtnet list [-h]

Output

The output has two main sections. The first section wrapped by the controller are global configurations and capabilities.

Each device has its own section under devices.

Example

The following is an example of a list with 1 static PF created:

# virtnet list
{
  "controller": {
    "emulation_manager": "mlx5_0",
    "max_hotplug_devices": "0",
    "max_virt_net_devices": "1",
    "max_virt_queues": "256",
    "max_tunnel_descriptors": "6",
    "supported_features": {
      "value": "0x8b00037700ef982f",
      "    0": "VIRTIO_NET_F_CSUM",
      "    1": "VIRTIO_NET_F_GUEST_CSUM",
      "    2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
      "    3": "VIRTIO_NET_F_MTU",
      "    5": "VIRTIO_NET_F_MAC",
      "   11": "VIRTIO_NET_F_HOST_TSO4",
      "   12": "VIRTIO_NET_F_HOST_TSO6",
      "   15": "VIRTIO_NET_F_MRG_RXBUF",
      "   16": "VIRTIO_NET_F_STATUS",
      "   17": "VIRTIO_NET_F_CTRL_VQ",
      "   18": "VIRTIO_NET_F_CTRL_RX",
      "   19": "VIRTIO_NET_F_CTRL_VLAN",
      "   21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
      "   22": "VIRTIO_NET_F_MQ",
      "   23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
      "   32": "VIRTIO_F_VERSION_1",
      "   33": "VIRTIO_F_IOMMU_PLATFORM",
      "   34": "VIRTIO_F_RING_PACKED",
      "   36": "VIRTIO_F_ORDER_PLATFORM",
      "   37": "VIRTIO_F_SR_IOV",
      "   38": "VIRTIO_F_NOTIFICATION_DATA",
      "   40": "VIRTIO_F_RING_RESET",
      "   41": "VIRTIO_F_ADMIN_VQ",
      "   56": "VIRTIO_NET_F_HOST_USO",
      "   57": "VIRTIO_NET_F_HASH_REPORT",
      "   59": "VIRTIO_NET_F_GUEST_HDRLEN",
      "   63": "VIRTIO_NET_F_SPEED_DUPLEX"
    },
    "supported_virt_queue_types": {
      "value": "0x1",
      "    0": "SPLIT"
    },
    "supported_event_modes": {
      "value": "0x5",
      "    0": "NO_MSIX_MODE",
      "    2": "MSIX_MODE"
    }
  },
  "devices": [
    {
      "pf_id": 0,
      "function_type": "static PF",
      "transitional": 0,
      "vuid": "MT2306XZ00BNVNETS0D0F2",
      "pci_bdf": "e2:00.2",
      "pci_vhca_id": "0x2",
      "pci_max_vfs": "0",
      "enabled_vfs": "0",
      "msix_num_pool_size": 0,
      "min_msix_num": 0,
      "max_msix_num": 256,
      "min_num_of_qp": 0,
      "max_num_of_qp": 127,
      "qp_pool_size": 0,
      "num_msix": "256",
      "num_queues": "255",
      "enabled_queues": "0",
      "max_queue_size": "256",
      "msix_config_vector": "0xFFFF",
      "mac": "16:B0:E0:41:B8:0D",
      "link_status": "1",
      "max_queue_pairs": "127",
      "mtu": "1500",
      "speed": "100000",
      "rss_max_key_size": "0",
      "supported_hash_types": "0x0",
      "ctrl_mac": "00:00:00:00:00:00",
      "ctrl_mq": "0",
      "sf_num": 1000,
      "sf_parent_device": "mlx5_0",
      "sf_parent_device_pci_addr": "0000:03:00.0",
      "sf_rep_net_device": "en3f0pf0sf1000",
      "sf_rep_net_ifindex": 10,
      "sf_rdma_device": "mlx5_3",
      "sf_cross_mkey": "0x12642",
      "sf_vhca_id": "0x124",
      "sf_rqt_num": "0x0",
      "aarfs": "disabled",
      "dim": "disabled"
    }
  ]
}

Query

This command queries detailed information for a given device, including all VQ information if created.

Syntax

virtnet query [-h] {[-a] | [-p PF] [-v VF] | [-u VUID]} [--dbg_stats] [-b] [--latency_stats] [-q QUEUE_ID] [--stats_clear] [rx_drops [status|--drops-only]]

Output

Output has two main sections.

• The first section, wrapped by devices, are configuration and capabilities on the device level, the majority of which are the same as the list command. This section only covers the differences between the two.

  Entry	Type	Description
  devices	String	Entries under this section is per-device information
  pci_dev_id	String	Virtio-net PCIe device ID. Default: 0x1041.  This option will be depreciated in the future.
  pci_vendor_id	String	Virtio-net PCIe vendor ID. Default: 0x1af4. This option will be depreciated in the future.
  pci_class_code	String	Virtio-net PCIe device class code. Default: 0x20000.  This option will be depreciated in the future.
  pci_subsys_id	String	Virtio-net PCIe vendor ID. Default: 0x1041. This option will be depreciated in the future.
  pci_subsys_vendor_id	String	Virtio-net PCIe subsystem vendor ID. Default: 0x1af4.  This option will be depreciated in the future.
  pci_revision_id	String	Virtio-net PCIe revision ID. Default: 1. This option will be depreciated in the future.
  device_features	String	Enabled device feature bits according to the virtio spec. Refer to section " DOCA Virtio-net Service Guide | Feature Bits ".
  driver_features	String	Enabled driver feature bits according to the virtio spec. Valid only when the driver probes the device. Refer to " DOCA Virtio-net Service Guide | Feature Bits ".
  status	String	Device status field bit masks according to the virtio spec: ACKNOWLEDGE (bit 0) DRIVER (bit 1) DRIVER_OK (bit 2) FEATURES_OK (bit 3) DEVICE_NEEDS_RESET (bit 6) FAILED (bit 7)
  reset	Number	Shows if the current virtio-net device undergoing reset: 0 – not undergoing reset 1 – undergoing reset
  enabled	Number	Shows if the current virtio-net device is enabled: 0 – disabled, likely FLR has occurred 1 – enabled

  
  

• The second section, wrapped by enabled-queues-info, provides per-VQ information:

  Entry	Type	Description
  index	Number	VQ index starting from 0 to enabled_queues
  size	Number	Driver VQ depth in bytes. It is bound by device max_queues_size.
  msix_vector	Number	The MSI-X vector number used for this VQ
  enable	Number	If current VQ is enabled or not 0 – disabled 1 – enabled
  notify_offset	Number	Driver reads this to calculate the offset from start of notification structure at which this virtqueue is located
  descriptor_address	Number	The physical address of the descriptor area
  driver_address	Number	The physical address of the driver area
  device_address	Number	The physical address of the device area
  received_desc	Number	Total number of received descriptors by the device on this VQ This option will be depreciated in the future.
  completed_desc	Number	Total number of completed descriptors by the device on this VQ This option will be depreciated in the future.
  bad_desc_errors	Number	Total number of bad descriptors received on this VQ This option will be depreciated in the future.
  error_cqes	Number	Total number of error CQ entries on this VQ This option will be depreciated in the future.
  exceed_max_chain	Number	Total number of chained descriptors received that exceed the maximum allowed chain by device This option will be depreciated in the future.
  invalid_buffer	Number	Total number of times the device tried to read or write buffer that is not registered to the device This option will be depreciated in the future.
  batch_number	Number	The number of RX descriptors for the last received packet. Relevant for BlueField-3 only. This option will be depreciated in the future.
  dma_q_used_number	Number	The DMA q index used for this VQ. Relevant for BlueField-3 only. This option will be depreciated in the future.
  handler_schd_number	Number	Scheduler number for this VQ. Relevant for BlueField-3 only. This option will be depreciated in the future.
  aux_handler_schd_number	Number	Aux scheduler number for this VQ. Relevant for BlueField-3 only. This option will be depreciated in the future.
  max_post_desc_number	Number	Maximum number of posted descriptors on this VQ. Relevant for DPA. This option will be depreciated in the future.
  total_bytes	Number	Total number of bytes handled by this VQ. Relevant for BlueField-3 only This option will be depreciated in the future.
  rq_cq_max_count	Number	Event generation moderation counter of the queue. Relevant for RQ. This option will be depreciated in the future.
  rq_cq_period	Number	Event generation moderation timer for the queue in 1µsec granularity. Relevant for RQ. This option will be depreciated in the future.
  rq_cq_period_mode	Number	Current period mode for RQ 0x0 – default_mode – use device best defaults 0x1 – upon_event – queue_period timer restarts upon event generation 0x2 – upon_cqe – queue_period timer restarts upon completion generation This option will be depreciated in the future.

Example

The following is an example of querying the information of the first PF:

# virtnet query -p 0
{
  "devices": [
    {
      "pf_id": 0,
      "function_type": "static PF",
      "transitional": 0,
      "vuid": "MT2349X00018VNETS0D0F1",
      "pci_bdf": "23:00.1",
      "pci_vhca_id": "0x1",
      "pci_max_vfs": "0",
      "enabled_vfs": "0",
      "pci_dev_id": "0x1041",
      "pci_vendor_id": "0x1af4",
      "pci_class_code": "0x20000",
      "pci_subsys_id": "0x1041",
      "pci_subsys_vendor_id": "0x1af4",
      "pci_revision_id": "1",
       "device_feature": {
        "value": "0x8930032300ef182f",
        "    0": "VIRTIO_NET_F_CSUM",
        "    1": "VIRTIO_NET_F_GUEST_CSUM",
        "    2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
        "    3": "VIRTIO_NET_F_MTU",
        "    5": "VIRTIO_NET_F_MAC",
        "   11": "VIRTIO_NET_F_HOST_TSO4",
        "   12": "VIRTIO_NET_F_HOST_TSO6",
        "   16": "VIRTIO_NET_F_STATUS",
        "   17": "VIRTIO_NET_F_CTRL_VQ",
        "   18": "VIRTIO_NET_F_CTRL_RX",
        "   19": "VIRTIO_NET_F_CTRL_VLAN",
        "   21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
        "   22": "VIRTIO_NET_F_MQ",
        "   23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
        "   32": "VIRTIO_F_VERSION_1",
        "   33": "VIRTIO_F_IOMMU_PLATFORM",
        "   37": "VIRTIO_F_SR_IOV",
        "   40": "VIRTIO_F_RING_RESET",
        "   41": "VIRTIO_F_ADMIN_VQ",
        "   52": "VIRTIO_NET_F_VQ_NOTF_COAL",
        "   53": "VIRTIO_NET_F_NOTF_COAL",
        "   56": "VIRTIO_NET_F_HOST_USO",
        "   59": "VIRTIO_NET_F_GUEST_HDRLEN",
        "   63": "VIRTIO_NET_F_SPEED_DUPLEX"
      },
      "driver_feature": {
        "value": "0x8000002300ef182f",
        "    0": "VIRTIO_NET_F_CSUM",
        "    1": "VIRTIO_NET_F_GUEST_CSUM",
        "    2": "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
        "    3": "VIRTIO_NET_F_MTU",
        "    5": "VIRTIO_NET_F_MAC",
        "   11": "VIRTIO_NET_F_HOST_TSO4",
        "   12": "VIRTIO_NET_F_HOST_TSO6",
        "   16": "VIRTIO_NET_F_STATUS",
        "   17": "VIRTIO_NET_F_CTRL_VQ",
        "   18": "VIRTIO_NET_F_CTRL_RX",
        "   19": "VIRTIO_NET_F_CTRL_VLAN",
        "   21": "VIRTIO_NET_F_GUEST_ANNOUNCE",
        "   22": "VIRTIO_NET_F_MQ",
        "   23": "VIRTIO_NET_F_CTRL_MAC_ADDR",
        "   32": "VIRTIO_F_VERSION_1",
        "   33": "VIRTIO_F_IOMMU_PLATFORM",
        "   37": "VIRTIO_F_SR_IOV",
        "   63": "VIRTIO_NET_F_SPEED_DUPLEX"
      },
      "status": {
        "value": "0xf",
        "    0": "ACK",
        "    1": "DRIVER",
        "    2": "DRIVER_OK",
        "    3": "FEATURES_OK"
      },
      "reset": "0",
      "enabled": "1",
      "num_msix": "64",
      "num_queues": "63",
      "enabled_queues": "63",
      "max_queue_size": "256",
      "msix_config_vector": "0x0",
      "mac": "4E:6A:E1:41:D8:BE",
      "link_status": "1",
      "max_queue_pairs": "31",
      "mtu": "1500",
      "speed": "200000",
      "rss_max_key_size": "0",
      "supported_hash_types": "0x0",
      "ctrl_mac": "4E:6A:E1:41:D8:BE",
      "ctrl_mq": "31",
      "sf_num": 1000,
      "sf_parent_device": "mlx5_0",
      "sf_parent_device_pci_addr": "0000:03:00.0",
      "sf_rep_net_device": "en3f0pf0sf1000",
      "sf_rep_net_ifindex": 12,
      "sf_rdma_device": "mlx5_2",
      "sf_cross_mkey": "0xC042",
      "sf_vhca_id": "0x7E8",
      "sf_rqt_num": "0x0",
      "aarfs": "disabled",
      "dim": "disabled",
      "enabled-queues-info": [
        {
          "index": "0",
          "size": "256",
          "msix_vector": "0x1",
          "enable": "1",
          "notify_offset": "0",
          "descriptor_address": "0x10cece000",
          "driver_address": "0x10cecf000",
          "device_address": "0x10cecf240",
          "received_desc": "256",
          "completed_desc": "0",
          "bad_desc_errors": "0",
          "error_cqes": "0",
          "exceed_max_chain": "0",
          "invalid_buffer": "0",
          "batch_number": "64",
          "dma_q_used_number": "6",
          "handler_schd_number": "4",
          "aux_handler_schd_number": "3",
          "max_post_desc_number": "0",
          "total_bytes": "0",
          "rq_cq_max_count": "0",
          "rq_cq_period": "0",
          "rq_cq_period_mode": "1"
        },
        ......
        }
      ]
    }
  ]
}

Stats

This command retrieves the packet counters for a specified device, including detailed information for all Rx and Tx virtqueues (VQs).

To enable/disable byte wise packet counters for each Rx queue, use the following command:

virtnet modify {[-p PF] [-v VF]} device -pkt_cnt {enable,disable}

• When enabled, byte-wise packet counters are initialized to zero.

• When disabled, the previous values are retained for debugging purposes. The command will still return these old, disabled counter values.

Syntax

virtnet stats [-h] {[-p PF] [-v VF] | [-u VUID]} [-q QUEUE_ID]

Output

The output has two sections.

• The first section wrapped by device are device details along with the packet counter statics enable state.

  Entry	Type	Description
  device	String	Entries under this section is per-device information
  pf_id	String	Physical function ID
  packet_counters	String	Indicates whether the packet counters feature is enabled or disabled

• The second section wrapped by queues-stats are information for each receive VQ.

  Entry	Type	Description
  VQ Index	Number	The VQ index starts at 0 (the first RQ) and continues up to the last SQ
  rx_64_or_less_octet_packets	Number	The number of packets received with a size of 0 to 64 bytes. Relevant for BlueField-3 RQ.
  rx_65_to_127_octet_packets	Number	The number of packets received with a size of 65 to 127 bytes. Relevant for BlueField-3 RQ.
  rx_128_to_255_octet_packets	Number	The number of packets received with a size of 128 to 255 bytes. Relevant for BlueField-3 RQ.
  rx_256_to_511_octet_packets	Number	The number of packets received with a size of 256 to 511 bytes. Relevant for BlueField-3 RQ.
  rx_512_to_1023_octet_packets	Number	The number of packets received with a size of 512 to 1023 bytes. Relevant for BlueField-3 RQ.
  rx_1024_to_1522_octet_packets	Number	The number of packets received with a size of 1024 to 1522 bytes. Relevant for BlueField-3 RQ.
  rx_1523_to_2047_octet_packets	Number	The number of packets received with a size of 1523 to 2047 bytes. Relevant for BlueField-3 RQ.
  rx_2048_to_4095_octet_packets	Number	The number of packets received with a size of 2048 to 4095 bytes. Relevant for BlueField-3 RQ.
  rx_4096_to_8191_octet_packets	Number	The number of packets received with a size of 4096 to 8191 bytes. Relevant for BlueField-3 RQ.
  rx_8192_to_9022_octet_packets	Number	The number of packets received with a size of 8192 to 9022 bytes. Relevant for BlueField-3 RQ.
  received_desc	Number	Total number of received descriptors by the device on this VQ
  completed_desc	Number	Total number of completed descriptors by the device on this VQ
  bad_desc_errors	Number	Total number of bad descriptors received on this VQ
  error_cqes	Number	Total number of error CQ entries on this VQ
  exceed_max_chain	Number	Total number of chained descriptors received that exceed the max allowed chain by device
  invalid_buffer	Number	Total number of times the device tried to read or write a buffer which is not registered to the device
  batch_number	Number	The number of RX descriptors for the last received packet. Relevant for BlueField-3.
  dma_q_used_number	Number	The DMA q index used for this VQ. Relevant for BlueField-3.
  handler_schd_number	Number	Scheduler number for this VQ. Relevant for BlueField-3.
  aux_handler_schd_number	Number	Aux scheduler number for this VQ. Relevant for BlueField-3.
  max_post_desc_number	Number	Maximum number of posted descriptors on this VQ. Relevant for DPA.
  total_bytes	Number	Total number of bytes handled by this VQ. Relevant for BlueField-3.
  rq_cq_max_count	Number	Event generation moderation counter of the queue. Relevant for RQ.
  rq_cq_period	Number	Event generation moderation timer for the queue in 1µsec granularity. Relevant for RQ.
  rq_cq_period_mode	Number	Current period mode for RQ 0x0 – default_mode – use device best defaults 0x1 – upon_event – queue_period timer restarts upon event generation 0x2 – upon_cqe – queue_period timer restarts upon completion generation

Example

The following is an example of querying the packet statistics information of PF 0 and VQ 0 (i.e., RQ):

# virtnet stats -p 0 -q 0
{'pf': '0x0', 'queue_id': '0x0'}
{
  "device": {
    "pf_id": 0,
    "packet_counters": "Enabled",
    "queues-stats": [
      {
        "VQ Index": 0,
        "rx_64_or_less_octet_packets": 0,
        "rx_65_to_127_octet_packets": 259,
        "rx_128_to_255_octet_packets": 0,
        "rx_256_to_511_octet_packets": 0,
        "rx_512_to_1023_octet_packets": 0,
        "rx_1024_to_1522_octet_packets": 0,
        "rx_1523_to_2047_octet_packets": 0,
        "rx_2048_to_4095_octet_packets": 199,
        "rx_4096_to_8191_octet_packets": 0,
        "rx_8192_to_9022_octet_packets": 0,
        "received_desc": "4096",
        "completed_desc": "0",
        "bad_desc_errors": "0",
        "error_cqes": "0",
        "exceed_max_chain": "0",
        "invalid_buffer": "0",
        "batch_number": "64",
        "dma_q_used_number": "0",
        "handler_schd_number": "44",
        "aux_handler_schd_number": "43",
        "max_post_desc_number": "0",
        "total_bytes": "0",
        "err_handler_schd_num": "0",
        "rq_cq_max_count": "0",
        "rq_cq_period": "0",
        "rq_cq_period_mode": "1"
      }
    ]
  }
}

Modify Device

This command modifies the attributes of a given device.

Syntax

The modify command supports three subcommands: device, queue, and global.

virtnet modify [-h] [-p PF] [-v VF] [-u VUID] [-a] {device,queue,global} ...

Device Options

virtnet modify device [-h] [-m MAC] [-t MTU] [-e SPEED] [-l LINK]
                           [-s STATE] [-f FEATURES]
                           [-o SUPPORTED_HASH_TYPES] [-k RSS_MAX_KEY_SIZE]
                           [-r RX_MODE] [-n MSIX_NUM] [-q MAX_QUEUE_SIZE]
                           [-b RX_DMA_Q_NUM] [-dc {enable,disable}]
                           [-pkt_cnt {enable,disable}] [-aarfs {enable,disable}]
                           [-qp MAX_QUEUE_PAIRS] [-dim {enable,disable}]

Enabling and Disabling Virtio-net Features

Configuring virtio-net features involves verifying what the DPU supports, modifying the device configuration, and confirming that the host driver successfully negotiated the new features.

Verify Supported Features

Check the full list of features supported by the underlying controller using the virtnet list command. 

virtnet list

Example JSON output:

"supported_features": {
    "value":        "0x8b00037700ef982f",
    "    0":        "VIRTIO_NET_F_CSUM",
    "    1":        "VIRTIO_NET_F_GUEST_CSUM",
    "    2":        "VIRTIO_NET_F_CTRL_GUEST_OFFLOADS",
    "    3":        "VIRTIO_NET_F_MTU",
    "    5":        "VIRTIO_NET_F_MAC",
    "   11":        "VIRTIO_NET_F_HOST_TSO4",
    "   12":        "VIRTIO_NET_F_HOST_TSO6",
    "   15":        "VIRTIO_NET_F_MRG_RXBUF",
    "   34":        "VIRTIO_F_RING_PACKED",
    "   63":        "VIRTIO_NET_F_SPEED_DUPLEX"
}

Check Currently Enabled Device Features

Verify which features are currently enabled on your target device (e.g., Physical Function 0) by inspecting the device_feature section of the query output. 

virtnet query -p 0 -b

Example JSON output:

"pci_bdf": "0f:00.2",
"device_feature": {
    "value":        "0x8100032300ef982f",
    "    0":        "VIRTIO_NET_F_CSUM",
    "   32":        "VIRTIO_F_VERSION_1",
    "   33":        "VIRTIO_F_IOMMU_PLATFORM",
    "   63":        "VIRTIO_NET_F_SPEED_DUPLEX"
}

Modify Device Features

Once unbound, use the -f flag to enable, disable, or explicitly set the feature bitmask.

• To enable a feature (+): 

  # Syntax: virtnet modify -p <pf> device -f +<feature_name>
  virtnet modify -p 0 device -f +VIRTIO_F_RING_PACKED
  
  

• To disable a feature (-): 

  # Syntax: virtnet modify -p <pf> device -f -<feature_name>
  virtnet modify -p 0 device -f -VIRTIO_F_RING_PACKED
  

• To set an explicit feature vector (bitmask) – You can overwrite the entire feature vector using a hex mask. For example, to add VIRTIO_F_RING_PACKED (bit 34) to a base vector of 0x8100032300EF182F, you calculate the logical OR (0x8100032300EF182F | 0x400000000 = 0x8100032700EF182F) and apply it:

  # Syntax: virtnet modify -p <pf> device -f <features_bitmask>
  virtnet modify -p 0 device -f 0x8100032700EF182F
  

Verify Driver Negotiation

After modifying the features and rebonding the driver, you must verify that the host operating system successfully negotiated the new features.

A feature is only fully active if it appears in both the device_feature and driver_feature lists. 

virtnet query -p 0 -b

Example output confirming VIRTIO_F_RING_PACKED (bit 34) was successfully negotiated:

"pci_bdf": "0f:00.2",

"device_feature": {
    "   34":        "VIRTIO_F_RING_PACKED"
},
"driver_feature": {
    "   34":        "VIRTIO_F_RING_PACKED"
}

Queue Options

virtnet modify queue [-h] -e {event,cqe} -n PERIOD -c MAX_COUNT

Global Options

virtnet modify global dc -mode <async|sync> --poll_freq <seconds>

Global output:

Global examples:

• Enable async mode with default polling interval:

  [dpu]# virtnet modify global dc -mode=async
  {
      "dc_mode": "async",
      "poll_freq_sec": 5
  }
  

• Enable async mode with custom polling interval:

  [dpu]# virtnet modify global dc -mode=async -poll_freq 10
  {
      "dc_mode": "async",
      "poll_freq_sec": 10
  }
  

• Disable async mode (return to sync):

  [dpu]# virtnet modify global dc -mode=sync
  {
      "dc_mode": "sync"
  }
  
  

  The async mode setting is persisted across controller restarts. No manual action is needed to restore it after a restart.

Output

Example

To modify the link status of the first VF on the first PF to be down:

# virtnet modify -p 0 device -l 0
{'pf': '0x0', 'all': '0x0', 'subcmd': '0x0', 'link': '0x0'}
{
  "errno": 0,
  "errstr": "Success"
}

Log

This command manages the log level of virtio-net-controller.

Syntax

virtnet log [-h] -l {info,err,debug}

Output

Example

To change the log level to info:

# virtnet log -l info
{'level': 'info'}
"Success"

To monitor current log output of the controller service with the latest 100 lines printed out:

$ journalctl -u virtio-net-controller -f -n 100

Validate

This command validates configurations of virtio-net-controller.

Syntax

virtnet validate [-h] -f PATH_TO_FILE

Output

Example

To check if virtnet.conf is a valid JSON file:

# virtnet validate -f /opt/mellanox/mlnx_virtnet/virtnet.conf
/opt/mellanox/mlnx_virtnet/virtnet.conf is valid

Version

This command prints current and updated version of virtio-net-controller.

Syntax

virtnet version [-h]

Output

Example

Check current and next available controller version:

# virtnet version
[
  {
    "Original Controller": "v24.10.17"
  },
  {
    "Destination Controller": "v24.10.19"
  }
]

Update

This command performs a live update to another version installed on the OS. Instead of a complete shutdown and recreating all existing devices, this procedure updates to the new version with minimal down time.

Syntax

virtnet update [-h] [-s | -t]

Output

Example

To start the live update process, run:

# virtnet update -s
{'start': '0x1'}
"Update started, use 'virtnet update -t' or check logs for status"

To check the update status during the update process:

# virtnet update -t
{'status': '0x1'}
{
  "status": "inactive",
  "last live update status": "success",
  "time_used (s)": 0.604152
}

Restart

This command performs a fast restart of the virtio-net-controller service. Compared to regular restart (using systemctl restart virtio-net-controller) this command has shorter down time per device.

Syntax

virtnet restart [-h]

Output

Example

To start the live update process, run:

# virtnet restart
SUCCESS

Health

This command shows health information for given devices.

Syntax

virtnet health [-h] {[-a] | [-p PF] [-v VF] | [-u VUID]} [show]

Output

Example

The following is an example of showing the information of the first PF:

# virtnet health -p 0 show
{'pf': '0x0', 'all': '0x0', 'subcmd': '0x0'}
{
  "pf_id": 0,
  "type": "static PF",
  "vuid": "MT2306XZ00BPVNETS0D0F1",
  "dev_status": {
    "value": "0xf",
    "    0": "ACK",
    "    1": "DRIVER",
    "    2": "DRIVER_OK",
    "    3": "FEATURES_OK"
  },
  "health_status": "Good",
  "health_recover_counter": 0,
  "dev_health_details": {
    "control_plane_errors": {
      "sf_rqt_update_err": 0,
      "sf_drop_create_err": 0,
      "sf_tir_create_err": 0,
      "steer_rx_domain_err": 0,
      "steer_rx_table_err": 0,
      "sf_flows_apply_err": 0,
      "aarfs_flow_init_err": 0,
      "vlan_flow_init_err": 0,
      "drop_cnt_config_err": 0
    },
    "data_plane_errors": {
      "sq_stall": 0,
      "dma_q_stall": 0,
      "spurious_db_invoke": 0,
      "aux_not_invoked": 0,
      "dma_q_errors": 0,
      "host_read_errors": 0
    }
  }
}

Error Code

CLI commands will return non-zero error code upon failure. All error numbers are negative. When an error occurs from the log, it could return an error number as well.

If the error number is greater than -1000, it is standard error. Please refer to Linux error code at errno

If the error number is less or equal -1000, please refer to the table below for the explanation.

Debug

The virtnet_cli tool provides debug commands for the event publisher.

• To view publisher configuration: 

  virtnet debug vnet_event config

• To view publisher counters:

  virtnet debug vnet_event stats

  CounterMeaningenqueuedTotal events successfully enqueued to the publish ring buffer.dropped_queue_fullEvents dropped because the queue was at max_queue_depth.json_encode_failEvents that failed JSON serialization (should always be 0).transport_publish_failEvents that the worker thread failed to publish to NATS.reconnect_attemptsNumber of times the worker attempted to reconnect to the broker.last_errorLast negative errno from a failed operation (0 = no error).

• To enable verbose debug logging (runtime):

  virtnet debug vnet_event --log_level 1
  

  This enables per-event syslog traces showing the NATS subject, JSON preview, and publish outcome. Set back to 0 to disable.

Feature Guidance

Counters

Packet Statistics

To query the packet counters, use stats command.

[dpu]# virtnet stats [-h] {[-p PF] [-v VF] | [-u VUID]} [-q QUEUE_ID]

The following command queries PF 0 and VQ 0 (i.e., RQ):

[dpu]# virtnet stats -p 0 -q 0

Output:

# virtnet stats -p 0 -q 0
{'pf': '0x0', 'queue_id': '0x0'}
{
  "device": {
    "pf_id": 0,
    "packet_counters": "Enabled",
    "queues-stats": [
      {
        "VQ Index": 0,
        "rx_64_or_less_octet_packets": 0,
        "rx_65_to_127_octet_packets": 259,
        "rx_128_to_255_octet_packets": 0,
        "rx_256_to_511_octet_packets": 0,
        "rx_512_to_1023_octet_packets": 0,
        "rx_1024_to_1522_octet_packets": 0,
        "rx_1523_to_2047_octet_packets": 0,
        "rx_2048_to_4095_octet_packets": 199,
        "rx_4096_to_8191_octet_packets": 0,
        "rx_8192_to_9022_octet_packets": 0,
        "received_desc": "4096",
        "completed_desc": "0",
        "bad_desc_errors": "0",
        "error_cqes": "0",
        "exceed_max_chain": "0",
        "invalid_buffer": "0",
        "batch_number": "64",
        "dma_q_used_number": "0",
        "handler_schd_number": "44",
        "aux_handler_schd_number": "43",
        "max_post_desc_number": "0",
        "total_bytes": "0",
        "err_handler_schd_num": "0",
        "rq_cq_max_count": "0",
        "rq_cq_period": "0",
        "rq_cq_period_mode": "1"
      }
    ]
  }
}

The output has two sections.

• The first section, wrapped by device, are device details along with the packet counter statics enable state.

  Entry	Type	Description
  device	String	Entries under this section is per device information
  pf_id	String	Physical function ID
  packet_counters	String	packet counters feature: enabled/disabled

• The second section, wrapped by queues-stats, are information for each receive VQ.

  Entry	Type	Description
  VQ Index	Number	The VQ index starts at 0 (the first RQ) and continues up to the last SQ
  rx_64_or_less_octet_packets	Number	The number of packets received with a size of 0 to 64 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_65_to_127_octet_packets	Number	The number of packets received with a size of 65 to 127 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_128_to_255_octet_packets	Number	The number of packets received with a size of 128 to 255 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_256_to_511_octet_packets	Number	The number of packets received with a size of 256 to 511 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_512_to_1023_octet_packets	Number	The number of packets received with a size of 512 to 1023 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_1024_to_1522_octet_packets	Number	The number of packets received with a size of 1024 to 1522 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_1523_to_2047_octet_packets	Number	The number of packets received with a size of 1523 to 2047 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_2048_to_4095_octet_packets	Number	The number of packets received with a size of 2048 to 4095 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_4096_to_8191_octet_packets	Number	The number of packets received with a size of 4096 to 8191 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  rx_8192_to_9022_octet_packets	Number	The number of packets received with a size of 8192 to 9022 bytes. Relevant for BlueField-3 RQ when packet counter is enabled.
  received_desc	Number	Total number of received descriptors by the device on this VQ
  completed_desc	Number	Total number of completed descriptors by the device on this VQ
  bad_desc_errors	Number	Total number of bad descriptors received on this VQ
  error_cqes	Number	Total number of errors CQ entries on this VQ
  exceed_max_chain	Number	Total number of chained descriptors received that exceed the max allowed chain by the device
  invalid_buffer	Number	Total number of times device tried to read or write buffer that is not registered to the device
  batch_number	Number	The number of RX descriptors for the last received packet. Relevant for BlueField-3.
  dma_q_used_number	Number	The DMA q index used for this VQ. Relevant for BlueField-3.
  handler_schd_number	Number	Scheduler number for this VQ. Relevant for BlueField-3.
  aux_handler_schd_number	Number	Aux scheduler number for this VQ. Relevant for BlueField-3.
  max_post_desc_number	Number	Maximum number of posted descriptors on this VQ. Relevant for DPA.
  total_bytes	Number	Total number of bytes handled by this VQ. Relevant for BlueField-3.
  rq_cq_max_count	Number	Event generation moderation counter of the queue. Relevant for RQ.
  rq_cq_period	Number	Event generation moderation timer for the queue in 1µsec granularity. Relevant for RQ.
  rq_cq_period_mode	Number	Current period mode for RQ 0x0 – default_mode – use device best defaults 0x1 – upon_event – queue_period timer restarts upon event generation 0x2 – upon_cqe – queue_period timer restarts upon completion generation The second section wrapped by queues-stats IS information for each receive VQ.

VQ Statistics

To query Rx VQ statistics, use the corresponding VQ index. For example, If there are 3 queues configured then to query Rx, VQ uses queue 0, Tx VQ uses queue 1, and Ctrl VQ uses queue 2.

The following is the command to query PF 0, VF 0, and VQ 0 (i.e., Rx).

[dpu]# virtnet query -p 0 -v 0 -q 0

Output:

"enabled-queues-info": [
  {
    "index": "0",
    "size": "256",
    "msix_vector": "0x1",
    "enable": "1",
    "notify_offset": "0",
    "descriptor_address": "0xffffe000",
    "driver_address": "0xfffff000",
    "device_address": "0xfffff240",
    "received_desc": "256",
    "completed_desc": "19",
    "bad_desc_errors": "0",
    "error_cqes": "0",
    "exceed_max_chain": "0",
    "invalid_buffer": "0",
    "batch_number": "64",
    "dma_q_used_number": "0",
    "handler_schd_number": "4",
    "aux_handler_schd_number": "3",
    "max_post_desc_number": "0",
    "total_bytes": "6460",
    "rq_cq_max_count": "0",
    "rq_cq_period": "0",
    "rq_cq_period_mode": "1"
  }

The following are some of the important VQ counters:

RQ Drop Counter

When DPA is the data path provider, each RQ has its corresponding drop counter, which counts the number of packets dropped inside the DPA virtio RQs. 

The drop counter only increments (initial value being 0), and its value gets reset to 0 when disabled.

Enabling/Disabling Drop Counters

RQ drop counter can be enabled and disabled per device as follows (using VF 0 on PF 0): 

[dpu]# virtnet modify -p 0 -v 0 device -dc enable
[dpu]# virtnet modify -p 0 -v 0 device -dc disable

Querying Drop Counters

Per-device Query

To query the drop counter value(s) for a specific device, run: 

[dpu]# virtnet query -p 0 -v 0 | grep num_desc_drop_pkts

If there is more than one RQ for a device, the drop count is the sum of all RQs' values.

Global Drop Counter Summary

To query the total drop count across all devices with a single command: 

virtnet query rx_drops

Output:

Examples:

• Query total drops across all devices: 

  [dpu]# virtnet query -p 0 -v 0 | grep num_desc_drop_pkts
  

  JSON output:

  {
      "packets": 98777777,
      "devices_with_drops": [
          { "pf": 0, "type": "PF", "packets": 12345 },
          { "pf": 0, "vf": 3, "type": "VF", "packets": 45678 },
          { "pf": 0, "vf": 17, "type": "VF", "packets": 23456 }
      ]
  }
  

• Query with no drops present:

  [dpu]# virtnet query -p 0 -v 0 | grep num_desc_drop_pkts
  

  JSON output:

  {
      "packets": 0,
      "devices_with_drops": []
  }
  

Per-device Total Query

To query the total drop count for a single PF or VF (sum of all its RQs):

virtnet query -p <PF> [-v <VF>] rx_drops

Output: