DOCA Virtio-net Emulation Application Guide

This guide describes the DOCA VNet PCI Device reference application in doca/applications/vnet_pci_dev.

1. Introduction

The DOCA VirtIO-net PCI Device Application emulates one or more virtio-net PCIe endpoints on NVIDIA BlueField DPUs. The application uses DOCA DevEmu PCI TLP APIs to emulate PCIe configuration/MMIO behavior and DOCA DevEmu VNet APIs to provide the hardware-backed VirtIO network datapath.

The host sees standard virtio-net PCI devices behind a software PCI switch topology. The DPU application owns PCI enumeration behavior, VirtIO PCI BAR emulation, MSI/MSI-X handling, hotplug signaling, queue lifecycle, control virtqueue handling, SR-IOV, and optional live update handover.

1.1. System Design

1.1.1. Host Side (Virtio-net Client Driver)

The host runs the standard Linux virtio_net driver.

Host responsibilities include:

• Enumerating the emulated PCI switch and VirtIO-net endpoints.

• Negotiating VirtIO feature bits.

• Programming VirtIO common config registers.

• Allocating and configuring RX/TX virtqueues.

• Configuring MSI-X vectors.

• Sending queue notifications through the notify BAR.

• Sending control virtqueue commands, especially multi-queue commands such as VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

The host network interface appears as a normal Linux NIC and can be inspected with tools such as ip link, ethtool, and lspci.

1.1.2. BlueField DPU (DOCA Virtio-net Emulation Application)

The DPU runs doca vnet_pci_dev.

The application:

• Creates a DOCA VNet PCI TLP device type.

• Creates a software PCI topology with one upstream port, several downstream ports (one downstream port per endpoint), and one VirtIO-net endpoint per downstream port.

• Handles PCIe Config Read/Write and MMIO Read/Write TLPs.

• Emulates VirtIO PCI modern capabilities and network device configuration.

• Creates one DOCA VNet offload engine per endpoint.

• Creates RX/TX virtqueue objects and an optional control virtqueue.

• Uses a PCI config workqueue and worker progress engines to keep the main TLP progress loop responsive.

• Supports static endpoint creation and runtime hotplug/unplug.

• Supports active/standby live update through --lu-mode.

1.1.3. Data Path Provider

Unlike the vblk refapp, the vnet refapp does not expose a --provider DPA|DPU flag.

The packet datapath is provided by DOCA DevEmu VNet offload and the underlying NIC/firmware path after the VirtIO queues and offload engine are configured. The application itself is mostly a PCI/VirtIO control-plane and lifecycle orchestrator. It does not implement a userspace packet copy loop for RX/TX data queues.

1.2. End-to-End Flow

device_status (host writes)   RESET(0)-> ACK(1)-> DRIVER(3)-> FEATURES_OK(0xB)-> DRIVER_OK(0xF)-> OPERATIONAL
                                                                |                    |
                              (DPU, async on FEATURES_OK)       |   (DPU, async on DRIVER_OK)
                              engine start; create RX/TX VQs;   |   apply VQ shadow + start VQs;
                              create CVQ IO ctx (if MQ)         |   bind CVQ; enable engine; stats
 host writes 0  -> device_status held at NEEDS_RESET(0x40) until async controller cleanup completes -> 0

1. The DPU application starts and creates the PCI TLP type, TLP channel, software PCI topology, and VirtIO-net device model.

2. In static mode, endpoints are created immediately. In hotplug mode, endpoints are created after plug <idx>.

3. The host enumerates the PCI topology and discovers VirtIO-net endpoints.

4. The host virtio_net driver reads features and device configuration such as MAC, MTU, link status, speed, duplex, and maximum queue pairs.

5. The driver writes VirtIO status bits: ACKNOWLEDGE, DRIVER, FEATURES_OK, and DRIVER_OK.

6. On FEATURES_OK, the DPU prepares the offload engine, creates VQ objects, and creates the CVQ IO context if multi-queue is negotiated.

7. On DRIVER_OK, the DPU applies queue configuration, starts VQs, binds CVQ, and enables the offload engine.

8. Packet RX/TX proceeds through the DOCA VNet offload datapath.

9. Runtime CVQ commands can adjust the number of active queue pairs.

10. Reset, hotplug, unplug, shutdown, and live update paths tear down or transfer state while preserving PCI/VirtIO correctness.

2. Application Architecture

2.1. Core Components

Key files:

• vnet_pci_dev.c: main(), logging, signal handling, arguments registration, top-level config.

• vnet_pci_dev_core.c: application orchestration, topology setup, controller lifecycle, VirtIO status callback, hotplug, MQ handling, progress loop.

• vnet_pci_device.c: PCI config workqueue, TLP channel callback, config-space emulation, MMIO emulation, VirtIO PCI device model.

• vnet_pci_device.h: public interfaces, BAR layout, queue limits, struct vnet_pci_device.

• vnet_pci_dev_core.h: struct vnet_pci_dev_config, struct vnet_pci_dev_controller, feature defaults.

• pci_spec_tlp.h: PCI/TLP constants, topology macros, struct tlp_context, struct pci_device_config.

• vnet_virtio_types.h: VirtIO-net structures, status bits, feature bits, CVQ command definitions.

• vnet_pci_dev_lu.c and vnet_pci_dev_lu.h: live update active/standby handover.

Important data structures:

• struct tlp_context: global TLP/PCI state, DOCA device, PCI type, TLP channel, PE1, topology, BDF map, ACG queue, VirtIO devices, VNet controllers, and runtime parameters.

• struct vnet_pci_device: one per endpoint; holds VirtIO common config, network config, feature state, VQ shadow array, reset generation, and status callback.

• struct vnet_pci_dev_controller: one per endpoint; owns the DOCA VNet offload engine, RX/TX VQ arrays, CVQ, VNet IO context, worker PE, counters/stats, and atomic lifecycle state.

• PCI config workqueue: moves slow operations away from the main TLP progress loop.

• PE1: main progress engine for TLP handling.

• PE2: per-controller worker progress engines used by the workqueue and CVQ IO contexts.

2.1.1. Virtio-net Device Model

The app models a modern VirtIO-net PCI endpoint.

Each endpoint exposes:

• VirtIO PCI vendor/device IDs.

• PCI class code for network controller.

• PCIe capability.

• MSI-X capability.

• VirtIO common config capability.

• VirtIO notify capability.

• VirtIO ISR capability.

• VirtIO device config capability.

• VirtIO PCI config access capability.

The network device config contains:

• MAC address.

• Link status.

• Maximum virtqueue pairs.

• MTU.

• Speed.

• Duplex mode.

• RSS-related size/hash fields.

The base MAC comes from --mac-addr. Per-endpoint MACs are generated by incrementing the last octet by the endpoint index.

2.1.2. Virtio Queues

RX VQ = qp*2   TX VQ = qp*2+1   CVQ = max_qp*2   Total VQs = 2*max_qp+1
default max_qp=8   hard max=127   initial active qp=1   queue size default 1024, 16..4096, power-of-two

Queue model:

• RX VQ index: queue_pair_index * 2.

• TX VQ index: queue_pair_index * 2 + 1.

• CVQ index: max_queue_pairs * 2.

• Total VQs: max_queue_pairs * 2 + 1.

Defaults and limits:

• Default max queue pairs: 8.

• Maximum queue pairs: 127.

• Initial active queue pairs: 1.

• Default queue size: 1024.

• Queue size range: 16-4096.

• Queue size must be a power of two.

The application maintains a shadow copy of host-programmed VQ config in struct vnet_pci_device. Worker threads snapshot this shadow state before applying it to DOCA VQs, avoiding races with reset.

2.1.3. PCIe TLP Emulation

The application handles PCIe TLPs directly.

TLP handling flow:

1. vnet_pci_dev_event_cb() receives a TLP channel request.

2. ACG requests are cached for later MSI Memory Write TLPs.

3. PCI event requests are completed directly.

4. Normal requests are classified as config read/write, memory read/write, or unsupported.

5. The target device is resolved through BDF lookup or BAR address lookup.

6. Endpoint TLPs are protected with an endpoint read lock (to avoid racing hot-unplug destruction).

7. Config-space and MMIO handlers generate the correct completions.

8. Invalid/unmapped requests return Unsupported Request through the dummy device path.

The app emulates both Type 0 endpoint config space and Type 1 bridge config space.

2.1.4. PCI Switch Topology

The topology is:

USP -+- DSP[0] - EP[0]      USP: upstream port (Type-1)   DSP: downstream port per EP (Type-1 + HP slot)
     +- DSP[N] - EP[N]      EP: virtio-net (Type-0)       DUMMY: Unsupported Request for invalid BDFs

DUMMY device for unsupported BDF routing

Components:

• USP: upstream switch port, root of the software switch.

• DSP: downstream switch port, one per endpoint, with PCIe hotplug slot capability.

• EP: VirtIO-net endpoint.

• DUMMY: returns Unsupported Request for invalid probes.

The host assigns bus/device/function numbers during enumeration. The app caches BDF mappings and updates them on bus renumbering.

2.1.5. SR-IOV

Each PF can expose up to 64 VFs (each VF an independent virtio-net device, usable on the host or passed to a VM).
One SF backs each VF (its offload-engine representor must be OVS-bridged for RX).
ARI places VFs at functions 1..N of the PF bus (first_vf_offset=1, vf_stride=1), so >7 VFs/PF work; VF Device ID 0x1041.
VF queue count mirrors the PF -q.
Per-VF MSI-X num = per-VF doorbells = 2 * max_qp + 2.
Enabled by --total-vfs.

2.1.6. Threading & latency model (core invariant)

PE1 (main thread): services host config/MMIO TLP completions, < 1 ms, MUST NOT BLOCK
    | submit heavy op
    v
PCI config workqueue (preallocated 1024-item pool)  ->  worker on PE2 (one per controller):
    engine start/enable | VQ create/config | CVQ IO-ctx | controller cleanup |
    hotplug create/destroy | MQ start (<=2 concurrent) | diagnostics/stats

Any operation that can block on firmware/DOCA transitions is moved off PE1, or the host's non-posted reads stall (Linux probe/reboot soft-lock).
Reset safety: per-device reset generations, cancel flags, VQ-shadow snapshots, bounded waits.

2.2. Initialization Process

The initialization process is:

1. Parse command line arguments.

2. Allocate and initialize tlp_context.

3. Open the DOCA device by IB device name or PCI address.

4. Create PE1 for TLP progress.

5. Configure optional CPU affinity.

6. Initialize the PCI config workqueue.

7. Verify VNet PCI TLP support.

8. Create the VNet PCI TLP type.

9. Attach the DOCA device to the PCI type.

10. Configure MSI-X count and doorbell count.

11. Start the PCI type.

12. Query and validate the predefined VirtIO-net BAR layout.

13. Create and configure the TLP channel.

14. Connect the TLP channel to PE1 and start it.

15. Initialize software PCI topology.

16. Initialize DOCA VNet subsystem.

17. Create VirtIO-net device models.

18. Create endpoints immediately in static mode, or defer endpoint creation in hotplug mode.

19. Allocate per-endpoint transaction regions.

20. Create one worker PE per controller.

21. Enter the progress loop.

2.3. Reset and Shutdown

Reset is triggered when the host writes device_status = 0.

The app:

• Marks the device as canceling in-progress worker operations.

• Increments the per-device reset generation.

• Reinitializes VirtIO queue shadow state.

• Holds device_status at NEEDS_RESET while controller cleanup is pending.

• Disables/stops/destroys VQs.

• Stops and destroys the CVQ IO context when needed.

• Keeps the offload engine object alive for normal driver rebind.

• Destroys the offload engine only during endpoint destruction or final shutdown.

Shutdown:

• Stops accepting new workqueue items.

• Waits for outstanding work and MQ-start threads.

• Destroys VirtIO device models.

• Destroys endpoint controllers, TLP devices, and representors.

• Flushes ACG credits.

• Stops and destroys the TLP channel.

• Stops/destroys PCI type and DOCA VNet subsystem.

• Destroys worker PEs and PE1.

• Frees tlp_context.

3. DOCA Libraries

The application uses:

• DOCA DevEmu PCI TLP: PCIe TLP channel, PCI type, TLP devices, ACG, MSI/MSI-X support.

• DOCA DevEmu VNet: VirtIO-net offload engine, RX/TX VQs, CVQ, counters.

• DOCA DevEmu VirtIO: generic VirtIO offload engine and VQ lifecycle APIs.

• DOCA PE: progress engines for TLP and worker-side asynchronous operations.

• DOCA Arg Parser: command-line parsing.

• DOCA Log: application and SDK logging.

• libibverbs: used by build and live update/device handling paths.

• Optional libbsd: used for strlcpy when available.

4. Dependencies

Required:

• BlueField platform with DOCA DevEmu VNet PCI TLP support.

• DOCA SDK including DevEmu support.

• libibverbs.

• Firmware/device support for VirtIO-net BAR layout.

• PCI switch/TLP emulation enabled in firmware.

The app checks that the VNet TLP type is supported and that the TLP channel exposes exactly one NV switch TLP DSP. If not, initialization fails and asks for firmware configuration adjustment.

Unlike the vblk refapp, this code does not create application-managed DMA buffer pools or hugepage-backed data buffers for packet data. Packet movement is handled by DOCA VNet offload.

5. Compiling the Application

5.1. Compiling All Applications

cd /opt/mellanox/doca/applications/
meson setup /tmp/build && ninja -C /tmp/build

5.2. Compiling Only the Current Application

cd /opt/mellanox/doca/applications/
meson setup /tmp/build -Denable_all_applications=false -Denable_vnet_pci_dev=true && ninja -C /tmp/build

Alternatively, edit applications/meson_options.txt:

• Set enable_all_applications to false.

• Set enable_vnet_pci_dev to true.

Then run the normal meson && ninja build.

5.3. Build Outputs

The build produces one executable:

doca_vnet_pci_dev

Typical output path:

/tmp/build/vnet_pci_dev/doca_vnet_pci_dev

Live update is implemented inside the same executable through --lu-mode; there are no separate *_lu and *_emu binaries in the current vnet app.

6. Running the Application

6.1. Prerequisites

A typical setup must enable TLP and PCI switch emulation in firmware. The code expects one NV switch TLP downstream port.

Example firmware configuration pattern:

mlxconfig -y -d /dev/mst/mt41692_pciconf0 reset
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_NUM_PF=32
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_ENABLE=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_ENABLE=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_TLP_PORT=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_PORT=2

# SR-IOV supported up to 256 VFs, thus need set 320 TOTAL_SF (256 + 32 + 32).

mlxconfig -y -d /dev/mst/mt41692_pciconf0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8 PF_TOTAL_SF=320

Cold-reboot is required after firmware configuration changes.

6.2. Application Usage

Usage: doca_vnet_pci_dev [DOCA Flags] [Program Flags]

Common DOCA flags include:

• -h, --help

• -v, --version

• -l, --log-level

• --sdk-log-level

• --log-filter

• -j, --json

Program flags are described in the Command Line Flags section below.

6.2.1. Example Command Line Execution

Run with defaults in hotplug mode:

./doca_vnet_pci_dev [-H 1]

Run in static mode with four endpoints and 16 max queue pairs:

./doca_vnet_pci_dev -n 4 -q 16 -H 0

Run with custom MAC, MTU, and speed:

./doca_vnet_pci_dev -m 52:54:00:12:34:56 -t 9000 -s 100000

Run with explicit CPU affinity:

./doca_vnet_pci_dev --tlp-core 8 --worker-core 9 --mq-core 10

Run with SR-IOV (4 PF x 4 VF x 4 QP):

./doca_vnet_pci_dev -H 0 -n 4 --total-vfs 4 -q 4

Run in live update active mode(-H 0 for static mode, -H 1 for hotplug mode):

./doca_vnet_pci_dev -H 0 --lu-mode active

Run a standby process for live update:

./doca_vnet_pci_dev -H 0 --lu-mode standby

Run with MRG_RXBUF and GUEST_CSUM features enabled (driver need to negotiate)

./doca_vnet_pci_dev -H 0 -t 9000 --mrg-rxbuf --guest-csum

6.2.2. Runtime Commands

Available through stdin:

• plug <DSP_IDX>: create and hotplug an endpoint into a DSP slot. Hotplug mode only.

• unplug <DSP_IDX>: request hot-unplug of the endpoint in a DSP slot. Hotplug mode only.

• speed <EP_IDX> <Mbps>: update the reported link speed and raise a config-change MSI-X notification.

Examples:

plug 0
unplug 0
speed 0 25000

6.2.3. Verifying the Device (Host Side)

After startup or hotplug:

lspci | grep -i virtio
ip link

Inspect queue support:

ethtool -l <interface>

Change active queue pairs:

ethtool -L <interface> combined 4

Inspect link information:

ethtool <interface>

6.3. Command Line Flags

6.3.1. General Flags

The app uses DOCA Arg Parser, so standard DOCA flags are available:

• -h, --help: print usage.

• -v, --version: print version.

• -l, --log-level: set application log level.

• --sdk-log-level: set SDK log level.

• --log-filter: filter logs by module.

• -j, --json: parse flags from JSON.

6.3.2. Program Flags

• -p, --pci-addr: DOCA device PCI address. Default: 0000:03:00.0.

• -d, --ibdev-name: IB device name. Overrides --pci-addr.

• -m, --mac-addr: base MAC address. Default: 52:54:00:12:34:56.

• -t, --mtu: MTU, range 68-9000. Default: 1500.

• -s, --speed: link speed in Mbps. Default: 100000.

• -D, --duplex: duplex mode, 0 half or 1 full. Default: 1.

• -q, --max-queue-pairs: maximum queue pairs, range 1-127. Default: 8.

• -z, --queue-size: queue size, power of two, range 16-4096. Default: 1024.

• -n, --num-ep: number of endpoints, range 1-32. Default: 1.

• -H, --hotplug-mode: 1 hotplug, 0 static. Default: 1.

• --tlp-core: CPU core for the main TLP progress thread, or -1 for auto.

• --worker-core: CPU core for the PCI config worker thread, or -1 for auto.

• --mq-core: CPU core for MQ start threads, or -1 for auto.

• --lu-mode: live update mode: none, active, or standby. Default: none.

Validation rules:

• --tlp-core must not equal --worker-core when both are explicitly set.

• --mq-core must not equal --tlp-core when both are explicitly set.

• --queue-size must be a power of two.

• --mac-addr last octet must leave enough room for per-endpoint incrementing.

6.3.3. Optional Features Flags

• --guest-csum Driver handles packets with partial checksum (VIRTIO_NET_F_GUEST_CSUM)

• --guest-tso4 Driver can receive TSOv4 (VIRTIO_NET_F_GUEST_TSO4)

• --guest-tso6 Driver can receive TSOv6 (VIRTIO_NET_F_GUEST_TSO6)

• --guest-ecn Driver can receive TSO with ECN (VIRTIO_NET_F_GUEST_ECN)

• --guest-ufo Driver can receive UFO (VIRTIO_NET_F_GUEST_UFO)

• --host-ecn Device can receive TSO with ECN (VIRTIO_NET_F_HOST_ECN)

• --host-ufo Device can receive UFO (VIRTIO_NET_F_HOST_UFO)

• --mrg-rxbuf Driver can merge receive buffers (VIRTIO_NET_F_MRG_RXBUF)

6.4. Troubleshooting

Common issues:

• VNet TLP type unsupported: the device or firmware does not expose VirtIO-net emulation support.

• Wrong TLP port count: the app expects exactly one NV switch TLP DSP.

• Host does not see devices: verify firmware settings, run PCI rescan, and confirm the app is in static mode or that plug <idx> was issued.

• Hotplug command rejected: host hotplug interrupt may not be enabled yet, or a previous unplug is still in progress.

• Queue startup takes time with high --max-queue-pairs: MQ expansion is intentionally deferred and backgrounded.

• RSS/hash CVQ commands fail: the app does not implement those CVQ commands.

• Host stalls during development changes: keep slow DOCA operations off PE1; the existing workqueue design is there to avoid PCIe completion timeouts.

7. Application Code Flow

7.1. Framework Initialization

7.1.1. Initialization Flow

1. main() initializes logging.

2. doca_argp_init() initializes argument parsing.

3. register_vnet_pci_dev_params() registers application-specific flags.

4. doca_argp_start() parses CLI/JSON input.

5. Signal handlers are installed for SIGINT and SIGTERM.

6. vnet_pci_dev_run() receives the parsed config.

7. init_tlp_context() allocates topology state.

8. find_doca_device() opens the DOCA device by IB name or PCI address.

9. init_progress_engine() creates PE1.

10. vnet_pci_device_configure_affinity() records affinity preferences.

11. init_virtio_network_device() initializes PCI/TLP, topology, VNet, VirtIO device state, and endpoint creation.

12. Worker PEs are created and attached to the PCI config workqueue.

13. run_progress_loop() starts processing TLPs, runtime commands, MSI retries, and LU triggers.

7.1.2. Key Points

• PE1 is reserved for TLP handling and must remain responsive.

• Heavy operations are submitted to the PCI config workqueue.

• Worker PEs are created per controller so multi-endpoint operations can progress independently.

• In LU standby mode, device and channel restore alter the normal open/start flow.

7.2. Resource Management

7.2.1. Resource Management Flow

Main resources:

• tlp_context

• DOCA device handle

• PE1 main progress engine

• VNet PCI TLP type

• TLP channel

• ACG credit queue

• software PCI device configs

• BDF map

• VirtIO device models

• VNet controller array

• per-controller worker PEs

• per-controller offload engine

• RX/TX/CVQ objects

• VNet IO context for CVQ

• stats/counters handles

• per-endpoint transaction region memory

• PCI config workqueue and work-item pool

Resource creation order:

1. Allocate tlp_context.

2. Initialize locks, maps, endpoint arrays, and controller arrays.

3. Open DOCA device.

4. Create PE1.

5. Initialize PCI config workqueue.

6. Create/start PCI type.

7. Create TLP channel.

8. Initialize topology and VNet subsystem.

9. Create VirtIO device models.

10. Create endpoints if static mode.

11. Allocate transaction regions.

12. Create worker PEs.

13. Create offload/VQ/IO resources lazily as the host reaches VirtIO states.

7.2.2. Key Points

• The workqueue uses a preallocated pool of 1024 work items to avoid hot-path allocation.

• MQ start concurrency is limited to 2 to avoid memory pressure and firmware command saturation.

• Reset generation fields prevent stale async work from affecting a post-reset device.

• Endpoint read/write locks protect TLP handlers from hotplug destruction races.

• Diagnostics run on the worker side, not PE1.

7.3. PCI Switch Topology and Device Configuration

7.3.1. PCI Switch Topology Flow

1. init_device_topology() initializes all bridge and endpoint config structures.

2. USP is configured as a Type 1 bridge with upstream-port PCIe capability.

3. Each DSP is configured as a Type 1 bridge with downstream-port and hotplug slot capabilities.

4. Each endpoint is configured as a Type 0 network device.

5. The dummy device is configured for unsupported request handling.

6. In static mode, create_all_devices() creates endpoint representors and TLP devices.

7. In hotplug mode, create_device() is called later from the hotplug work item.

8. Host enumeration assigns bus numbers and BDFs.

9. BDF mappings are cached and updated if the host renumbers buses.

Endpoint creation flow:

1. Create or reattach a representor.

2. Query representor VHCA ID.

3. Create the TLP device.

4. Start the TLP device.

5. Create the VNet controller and offload engine.

6. Mark endpoint present.

7.3.2. Key Points

• DSP count equals endpoint count.

• Each DSP supports PCIe hotplug slot behavior.

• Hotplug signaling uses attention button and slot status bits.

• MSI for hotplug is sent as a Memory Write TLP using ACG credits.

• If ACG credits are exhausted, MSI retry is scheduled.

• Static mode sets slots to powered-on and link-active at startup.

7.4. Virtio-net Controller

7.4.1. Controller Initialization Flow

Controller creation happens in vnet_pci_dev_vnet_controller_create():

1. Resolve endpoint PF index.

2. Associate controller with the per-endpoint VirtIO device.

3. Convert TLP device to PCI endpoint.

4. Create a VNet offload engine, or import one during LU standby.

5. Configure MTU.

6. Configure MAC address.

7. Configure total number of queues.

8. Initialize controller atomics and deferred MQ state.

9. Allocate RX and TX VQ pointer arrays.

10. Query/log SF representor information.

11. Wait for host FEATURES_OK before starting hardware preparation.

On FEATURES_OK:

1. Validate driver features vector.

2. Determine if VIRTIO_NET_F_CTRL_VQ and VIRTIO_NET_F_MQ were negotiated.

3. Set initial active queue pairs.

4. Submit offload engine start.

5. Submit VQ object creation.

6. Submit CVQ IO context creation.

On DRIVER_OK:

1. Submit start-and-enable work.

2. Enable optional features if negotiated.

3. Snapshot VQ shadow config.

4. Configure RX/TX/CVQ VQs.

5. Start VQs.

6. Bind CVQ to IO context.

7. Enable VirtIO offload engine.

8. Create stats/counter resources.

7.4.2. Virtio Queue Lifecycle State Machine

Conceptual state flow:

UNCREATED --FEATURES_OK--> CREATED --host cfg--> CONFIGURED --start--> STARTED --enable--> ENABLED
   ^                                                                                        |
 DESTROYED <- STOPPED <- DISABLED <----------------- reset / group-disable -----------------+

Lifecycle by phase:

• UNCREATED: no DOCA VQ object exists.

• CREATED: RX/TX/CVQ object has been created after FEATURES_OK.

• CONFIGURED: host-provided queue size, MSI-X vector, descriptor address, driver address, and device address have been applied.

• STARTED: doca_devemu_virtio_vq_start() has completed.

• ENABLED: offload engine or VQ group enable has made the queue operational.

• DISABLED: VQ group or engine disable has stopped host-visible processing.

• STOPPED: VQ stop completed.

• DESTROYED: VQ object destroyed.

7.4.3. Key Points

• RX/TX VQs are created for the configured maximum queue pairs.

• CVQ is created only when MQ is negotiated.

• Data queues may be deferred during multi-endpoint startup so enumeration can settle.

• Increasing queue pairs completes the CVQ request immediately, then starts extra queues in background.

• Decreasing queue pairs disables and stops extra queues synchronously.

• Worker readers use snapshots of VQ shadow config to avoid reset races.

7.5. I/O Request Processing

7.5.1. I/O Request Processing Flow

For packet data:

1. Host virtio_net driver posts RX buffers and TX descriptors.

2. Host notifies queues through the VirtIO notify region.

3. DOCA VNet offload processes packet datapath using the configured VQs.

4. The application does not copy packet payloads in userspace.

5. Completion and interrupt behavior are handled through the DOCA VNet/VirtIO offload path.

For CVQ control requests:

1. Host submits a control command on CVQ.

2. DOCA VNet forwards the request to the registered IO context.

3. Worker PE progresses the IO context.

4. vnet_pci_dev_ctrl_req_handler() receives cls and cmd.

5. MQ pair-set commands are handled by the app.

6. MAC/RX/VLAN classes are acknowledged as backend-handled.

7. Unsupported classes/commands return VIRTIO_NET_ERR.

8. The request is completed with doca_devemu_vnet_ctrl_req_complete().

7.5.2. Key Points

• RX/TX data path is hardware/offload driven.

• CVQ is the main application-visible I/O request path.

• MQ expansion avoids blocking the host by completing the CVQ command before long VQ startup.

• RSS/hash CVQ commands are not implemented by the app.

• The app’s most important responsibility is correct lifecycle sequencing, not packet processing.

7.6. Device Lifecycle Management

7.6.1. Device Lifecycle State Machine

Endpoint-level conceptual state:

Endpoint: INIT -> MODEL_CREATED -> ENDPOINT_CREATED -> FEATURES_OK_PREP -> DRIVER_OK_ENABLE -> OPERATIONAL -> RESETTING -> CLEANED_FOR_REBIND -> (rebind) | DESTROYED

Hotplug state:

Hotplug : ABSENT --plug--> PRESENT(link down) --Power-ON--> DLActive --> OPERATIONAL
             ^                                                              | unplug
          DESTROYED <- delayed destroy <- Power-OFF (or 30s timeout) <- UNPLUG_PENDING

7.6.2. Hotplug Management Flow

Plug:

1. Runtime command plug <idx> is parsed.

2. Workqueue receives hotplug work.

3. The endpoint representor, TLP device, and VNet controller are created.

4. DSP slot status sets attention button, presence detect changed, and presence detect state.

5. MSI is sent if ACG credit and host MSI setup are available.

6. Host pciehp notices the slot event.

7. Host powers on the slot.

8. TLP handler observes Slot Control Power ON and sets DLActive.

9. Host enumerates and probes the VirtIO-net endpoint.

Unplug:

1. Runtime command unplug <idx> is parsed.

2. Workqueue marks endpoint pending_unplug.

3. DSP slot status raises attention button.

4. MSI is sent or retried.

5. Host pciehp unbinds the driver and powers off the slot.

6. TLP handler observes Slot Control Power OFF.

7. Delayed destroy work is submitted.

8. Controller, TLP device, and representor are destroyed.

9. Pending flags are cleared.

If the host never powers off the slot, the main loop forces removal after UNPLUG_TIMEOUT_SEC, currently 30 seconds.

7.6.3. Key Points

• Hotplug creation is intentionally done on the workqueue because endpoint creation can take hundreds of milliseconds.

• PE1 must keep progressing TLPs during hotplug.

• Plug and unplug use PCIe hotplug slot semantics rather than ad hoc host signaling.

• Unplug waits for host power-off before destroying resources.

• A timeout safety net handles failed or stuck host removal.

7.7. SR-IOV

app init (per PF): pool_alloc_range -> create_pf_rep_ex(TotalVFs, stride=1, offset=1, vf_msix, vf_db)
                   -> PF tlp_dev -> SR-IOV cap shadow -> discover & open VF reps (templates only)

host: echo N > sriov_numvfs -> VF-Enable(VFE) 0->1 -> vfe_fan_out: per slot vnet_sriov_start_vf_slot (VF tlp_dev)

guest VF: FEATURES_OK -> vf_engine_create+start + vf_vqs_create + vf_io_context
          DRIVER_OK   -> vf_start_and_enable (start VQs, bind CVQ user_data=slot, enable engine)
          
per-VF FLR (DevCtl bit15): handle_vf_flr (coalesce) -> reset_vf_slot (tlp_dev stop->start; rep stays)

host: echo 0 -> VFE 1->0 -> stop_vf_slot (engine destroy + tlp_dev stop)

Limits: --total-vfs ≤ 64/PF (ARI ceiling 255); device-wide VF-VQ budget validated at init; VF cfg BARs read 0.
Mutually exclusive with LU.
Pass a VF to a VM via vfio-pci (1af4:1041) + QEMU/libvirt.

SR-IOV Support — one emulated PF exposes up to 64 independent virtio-net VFs (own cfg space, MSI-X, doorbells, engine, SF datapath); host-driven or VM-passthrough.
Off by default; --total-vfs <1..64>.
Split: vnet_sriov.c (pool/rep/tlp_dev), vnet_pci_device.c (cap + VFE fan-out + per-VF TLP/FLR), vnet_pci_dev_core.c (per-VF datapath).

7.7.1. 1 Model & rationale

SF-per-VF (must OVS-bridge each VF rep for RX); ARI functions 1..N (FirstVFOffset=1, VFStride=1), VF Device ID 0x1041; two-stage lazy bring-up (VFE creates tlp_dev only; engine deferred to guest FEATURES_OK, enabled at DRIVER_OK); deterministic per-VF MAC (base_low16 + num_ep + vf_index).

7.7.2. 2 Resources & validation

per VF: 1 GVMI, 2q+2 MSI-X/doorbells, 1 SF+rep, VNET_TOTAL_VQS(q)=2q+1 VQs. Table of vnet_sriov_validate_caps() checks → exact rejection strings (per-PF 64, ARI 255, device-wide budget 4352, aggregate 256, SDK pool/stride/msix). Fallback to non-rep_ex when pool_size==0.

7.7.3. 3 VF lifecycle

6-phase diagram (A init → B host VFE fan-out → C guest FEATURES_OK/DRIVER_OK → D per-VF FLR → E host disable → F exit), per-VF sub-state machine, a guard-atomics table (active, flr_in_progress, engine_started, vqs_created, engine_enabled, io_ctx_started/cvq_bound, num_active_qps), and the subtle fan-out semantics (VFE starts all TotalVFs; NumVFs write is shadow-only).

7.7.4. Per-VF FLR

A Function Level Reset resets a single VF in isolation, leaving its siblings and the parent PF running. The guest sets the Initiate-FLR bit in the VF's PCIe Device Control register; the TLP thread then runs vnet_pci_device_handle_vf_flr(), which:

1. atomically sets flr_in_progress, coalescing a duplicate FLR that arrives in the window (logged FLR coalesced);

2. resets the VF's config-space shadow synchronously, so the host reads a clean, post-reset config space immediately; and

3. queues VF_FLR_RESET, whose worker runs vnet_sriov_reset_vf_slot() = tlp_dev stop then start (the representor stays open), after which the guest re-probes through Phase C.

The worker always clears flr_in_progress on completion, even on error, so a VF never gets stuck coalescing. A PF-level FLR is handled separately by the PF's own virtio reset state machine.

7.7.5. SR-IOV PCIe Capability (Host View)

The PF presents a standard PCIe SR-IOV Extended Capability at config-space offset 0x100, chained 0x100 (SR-IOV) → 0x140 (ARI) → 0x148 (ACS); ARI enables VF function numbers above 7, and ACS gives per-VF IOMMU isolation for passthrough. Registers are synthesized on demand from struct vnet_sriov_pf_state (sriov_cap_read) and host writes are decoded back into it (sriov_cap_decode_write); the layout follows PCIe Base Spec (struct sriov_ext_cap_layout).

Setting VF-Enable triggers the fan-out described above.
A VF's own Type-0 BARs read 0 by design: a VF addresses its MMIO through the SR-IOV VF BAR aperture (DW9..14), as the specification requires.

7.7.6. VF datapath & VM passthrough

Because each VF has its own SF representor, its data-path must be bridged on the DPU before it carries traffic.
Add each VF representor to the OVS bridge (the per-VF engine-create log prints the ifindex to use):

ovs-vsctl add-port ovsbr1 <vf_representor>

To pass a VF through to a VM, bind it on the host to vfio-pci (IOMMU enabled) and attach it to the guest; the VF enumerates as PCI ID 1af4:1041:

QEMU:    -device vfio-pci,host=<VF BDF>
libvirt: <hostdev mode='subsystem' type='pci' managed='yes'> ... </hostdev>

Traffic path: guest -> VF -> DPU SF/representor -> OVS -> uplink.

7.7.7. Constraints & expected logs

• Stating that the host must set sriov_numvfs = 0 (and shut down any VM holding a VF) before rebooting, and that rebooting with VFE set leaves stale VF state on the DPU that can disrupt the next enumeration.

• Mutually exclusive with --lu-mode: live update transfers PF state only, so the app refuses to start with both enabled.

• Each VF's queue count mirrors the PF --max-queue-pairs.

• A PF cannot be hot-unplugged while VF-Enable is set or any VF datapath is live; the unplug is rejected (... (VFE=1, K live VF datapath(s)); disable it on the host first).

A healthy single-PF, single-VF bring-up produces this log sequence (match on text, not line numbers):

SR-IOV: dynamic SR-IOV supported (max_total_vfs=.., pool_size=..)
SR-IOV: pool reserved range [0, N)
SR-IOV: PF representor created with VF range [0, N) (vf_num_msix=18, vf_num_db=18)
SR-IOV: opened all N VF reps for this PF
SR-IOV VFE: 0 -> 1 (NumVFs=N, TotalVFs=N)
SR-IOV: VF[vf_index=g] tlp_dev started
SR-IOV VFE 0->1 fan-out: N/N slots started (0 failed)
SR-IOV: VF[slot=s, vf_index=g] FEATURES_OK - queueing engine bring-up (mq=1)
SR-IOV: VF[vf_index=g] offload engine created (idle, 17 VQs, MTU=1500)
SR-IOV: VF[vf_index=g] engine representor: vhca_id=0x.... ifindex=.. (add to OVS bridge for RX)
SR-IOV: VF[vf_index=g] created 8 queue pair(s) + CVQ
SR-IOV: VF[vf_index=g] CVQ IO context started
SR-IOV: VF[slot=s, vf_index=g] DRIVER_OK - queueing engine enable
SR-IOV: VF[vf_index=g] datapath operational (engine enabled, 8 queue pair(s))

7.8. Cleanup and Shutdown

7.8.1. Cleanup and Shutdown Flow

Shutdown begins when SIGINT or SIGTERM sets force_quit.

The main progress loop exits when:

• Quit is requested and no hot-unplug is pending.

• Hot-unplug pending has completed.

• Hot-unplug pending times out.

• LU handover is requested by SIGUSR1.

7.8.2. Phase 1 – Worker and Device Cleanup

The current vnet app does not have vblk-style per-IO worker threads named like vblk_io_ctx_thread(). Its equivalent first phase is workqueue and controller-side cleanup.

Steps:

1. Disable worker-side diagnostics.

2. Shut down PCI config workqueue.

3. Join outstanding MQ start threads.

4. Stop accepting new work.

5. Destroy VirtIO device models.

6. For each endpoint, destroy controller resources.

7. Disable VQs while engine is enabled.

8. Disable engine.

9. Stop VQs.

10. Destroy VQs.

11. Stop and destroy CVQ IO context.

12. Stop offload engine.

13. Destroy offload engine when required.

14. Destroy TLP devices and representors.

15. Clear endpoint present state.

7.8.3. Phase 2 – Reverse-Order Resource Cleanup

After workqueue shutdown:

1. Clear worker PE references from the workqueue.

2. Destroy per-controller worker PEs.

3. Flush pending ACG credits.

4. Stop the TLP channel context.

5. Drain PE1 until the TLP channel reaches idle or timeout.

6. Destroy the TLP channel.

7. Stop and destroy the PCI type.

8. Tear down DOCA VNet.

9. Remove the DOCA device from VNet.

10. Destroy PE1.

11. Free transaction regions, BDF maps, topology arrays, controller arrays, and VirtIO device arrays.

12. Close LU connections if any.

13. Destroy ARGP resources in main().

7.8.4. Key Points

• Cleanup order follows ownership dependencies.

• Workqueue shutdown happens before freeing objects referenced by work items.

• Worker PEs are destroyed only after the workqueue exits.

• TLP channel is stopped and drained before destruction.

• LU cleanup differs because the active process transfers traffic ownership before releasing local handles.

7.9. Live Upgrade (LU) Mode

The vnet refapp implements live update in the same binary through --lu-mode.

A(active) <- B(standby) connects -> A exports state + cmd FD -> B reconstructs device/EP/engines
from export blobs, replays VQ cfg (engines IDLE) -> B DEV_READY -> A DEV_GO -> B enables engines
-> B DEV_ACK -> TLP channel handover -> B active (chainable)

Modes:

• none: normal operation.

• active: current traffic owner.

• standby: process that restores state from active and takes over.

LU uses:

• Unix domain socket: /tmp/vnet_live_update.sock.

• Lock file: /tmp/vnet_live_update.lock.

• Shared memory: /vnet_live_update.

• TLP channel shared memory directory: /dev/shm/vnet_lu_channel.

• SIGUSR1 to trigger active handover.

High-level LU flow:

1. Active process runs and serves traffic.

2. Standby process connects to active.

3. Active sends command FD and exports device state.

4. Standby opens shared memory and reconstructs DOCA device state.

5. Standby recreates endpoint/controller/offload objects from export blobs.

6. Standby replays VirtIO config and VQ state without enabling engines.

7. Standby signals DEV_READY.

8. Active sends final DEV_GO.

9. Standby enables engines, one or more in parallel.

10. Standby sends DEV_ACK.

11. Channel LU transfers the TLP channel.

12. Standby becomes active and can support another chained LU.

Static LU state includes:

• endpoint VHCA ID

• MAC/MTU

• device features

• max queue pairs

• active queue pairs

• queue size

• MQ negotiation state

• per-VQ descriptor/driver/device addresses

• MSI-X vectors

• offload engine export blobs

• TLP channel configuration

Hotplug LU has additions:

• Number of populated slots

• Slot-presence bitmap

• Base MAC used to reconstruct per-slot MAC addresses

7.10. Crash Recovery

The current vnet refapp has live update state transfer but does not implement a separate persistent crash recovery mechanism comparable to a full crash-resume feature.

Important distinction:

• LU is coordinated and stateful: active and standby cooperate.

• Crash recovery would require reconstructing after unexpected termination without a live handover.

• The current code has robust cleanup, reset handling, timeout handling, and LU import/export paths, but no standalone crash recovery workflow is exposed as an application mode.

8. Virtio-net Operations Support Summary

Supported by the refapp/offload stack:

• RX/TX packet queues: supported through DOCA DevEmu VNet offload after VQs are configured and engine is enabled.

• Control virtqueue: supported when VIRTIO_NET_F_CTRL_VQ and VIRTIO_NET_F_MQ are negotiated.

• Multi-queue pair set: implemented for VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

• Runtime link speed update: supported through app command speed <idx> <Mbps> and config-change MSI-X.

• Hotplug/unplug: supported through PCIe hotplug slot emulation.

Partially supported or backend-acknowledged:

• MAC control class: acknowledged as backend-handled in the app callback.

• RX control class: acknowledged as backend-handled in the app callback.

• VLAN control class: acknowledged as backend-handled in the app callback.

Not implemented in the app callback:

• RSS configuration command.

• Hash configuration command.

• Unknown MQ commands.

• Unknown control classes.

9. Virtio Feature Bits Support Summary

Advertised by default:

• VIRTIO_NET_F_MAC: device reports MAC address.

• VIRTIO_NET_F_STATUS: link status is available.

• VIRTIO_NET_F_CSUM: checksum offload feature advertised.

• VIRTIO_NET_F_MTU: MTU is reported.

• VIRTIO_NET_F_SPEED_DUPLEX: speed and duplex are reported.

• VIRTIO_NET_F_HOST_TSO4: host can send TSOv4 packets to device.

• VIRTIO_NET_F_HOST_TSO6: host can send TSOv6 packets to device.

Always added by device initialization:

• VIRTIO_F_VERSION_1: modern VirtIO compliance.

• VIRTIO_F_ACCESS_PLATFORM: platform/IOMMU access semantics.

• VIRTIO_NET_F_CTRL_VQ: control virtqueue available.

• VIRTIO_NET_F_MQ: multi-queue support.

Defined but not part of the default advertised feature set unless code (SDK) is extended:
(some of the features are offered to enabled using CLI option)

• VIRTIO_NET_F_GUEST_CSUM

• VIRTIO_NET_F_CTRL_GUEST_OFFLOADS

• VIRTIO_NET_F_GUEST_TSO4

• VIRTIO_NET_F_GUEST_TSO6

• VIRTIO_NET_F_GUEST_ECN

• VIRTIO_NET_F_GUEST_UFO

• VIRTIO_NET_F_HOST_ECN

• VIRTIO_NET_F_HOST_UFO

• VIRTIO_NET_F_MRG_RXBUF

• VIRTIO_NET_F_CTRL_RX

• VIRTIO_NET_F_CTRL_VLAN

10. Developer Notes

The main engineering theme in this refapp is latency isolation. PE1 must continuously progress PCI TLP completions, so any operation that can block on firmware or DOCA state transitions is moved to the PCI config workqueue or detached MQ-start threads.

The second key theme is reset safety. The app uses reset generations, cancellation flags, VQ shadow snapshots, and short bounded waits so stale worker operations cannot corrupt a newly reset device.

The third key theme is host compatibility. Hotplug follows PCIe slot semantics, VirtIO status transitions follow the expected driver lifecycle, and CVQ MQ commands are completed quickly to avoid Linux probe and soft-lockup issues.

Last updated: September 01, 2026