Virtio Acceleration through Hardware vDPA

Hardware vDPA Installation

Hardware vDPA requires QEMU v2.12 (or with upstream 6.1.0) and DPDK v20.11 as minimal versions.

To install QEMU:

Clone the sources:

git clone https://git.qemu.org/git/qemu.git
cd qemu
git checkout v2.12

Build QEMU:

mkdir bin
cd bin
../configure --target-list=x86_64-softmmu --enable-kvm
make -j24

To install DPDK:

Clone the sources:

git clone git://dpdk.org/dpdk
cd dpdk
git checkout v20.11

Install dependencies (if needed):

yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel pandoc libibverbs libmlx5 libmnl-devel -y

Configure DPDK:

export RTE_SDK=$PWD
make config T=x86_64-native-linuxapp-gcc
cd build
sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g' .config
sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=\)n/\1y/g' .config

Build DPDK:
```
make -j
```
Build the vDPA application:
```
cd $RTE_SDK/examples/vdpa/
make -j
```

Hardware vDPA Configuration

To configure huge pages:

mkdir -p /hugepages
mount -t hugetlbfs hugetlbfs /hugepages
echo <more> > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
echo <more> > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages

To configure a vDPA VirtIO interface in an existing VM's xml file (using libvirt):

Open the VM's configuration XML for editing:
```
virsh edit <domain name>
```

Perform the following:

Change the top line to:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Assign a memory amount and use 1GB page size for huge pages (size must be the same as that used for the vDPA application), so that the memory configuration looks as follows.

<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<memoryBacking>
  <hugepages>
    <page size='1048576' unit='KiB'/>
  </hugepages>
</memoryBacking>

Assign an amount of CPUs for the VM CPU configuration, so that the vcpu and cputune configuration looks as follows:

<vcpu placement='static'>5</vcpu>
<cputune>
  <vcpupin vcpu='0' cpuset='14'/>
  <vcpupin vcpu='1' cpuset='16'/>
  <vcpupin vcpu='2' cpuset='18'/>
  <vcpupin vcpu='3' cpuset='20'/>
  <vcpupin vcpu='4' cpuset='22'/>
</cputune>

Set the memory access for the CPUs to be shared, so that the cpu configuration looks as follows:

<cpu mode='custom' match='exact' check='partial'>
  <model fallback='allow'>Skylake-Server-IBRS</model>
  <numa>
    <cell id='0' cpus='0-4' memory='8388608' unit='KiB' memAccess='shared'/>
  </numa>
</cpu>

Set the emulator in use to be the one built in step 2, so that the emulator configuration looks as follows:
```
<emulator><path to qemu executable></emulator>
```

Add a virtio interface using QEMU command line argument entries, so that the new interface snippet looks as follows:

<qemu:commandline>
  <qemu:arg value='-chardev'/>
  <qemu:arg value='socket,id=charnet1,path=/tmp/sock-virtio0'/>
  <qemu:arg value='-netdev'/>
  <qemu:arg value='vhost-user,chardev=charnet1,queues=16,id=hostnet1'/>
  <qemu:arg value='-device'/>
  <qemu:arg value='virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=e4:11:c6:d3:45:f2,bus=pci.0,addr=0x6,
   page-per-vq=on,rx_queue_size=1024,tx_queue_size=1024'/>
</qemu:commandline>

In this snippet, the vhostuser socket file path, the amount of queues, the MAC and the PCIe slot of the virtio device can be configured.

Running Hardware vDPA

Hardware vDPA supports switchdev mode only.

Create the ASAP² environment:
1. Create the VFs.
2. Enter switchdev mode.
3. Set up OVS.

Run the vDPA application:

cd $RTE_SDK/examples/vdpa/build
./vdpa -w <VF PCI BDF>,class=vdpa --log-level=pmd,info -- -i

Create a vDPA port via the vDPA application CLI:
```
create /tmp/sock-virtio0 <PCI DEVICE BDF>
```
The vhostuser socket file path must be the one used when configuring the VM.
Start the VM:
```
virsh start <domain name>
```

For further information on the vDPA application, visit the Vdpa Sample Application DPDK documentation.

Last updated: August 20, 2024