DOCA Virtio-net Emulation Application Guide

This guide describes the DOCA VNet PCI Device reference application in doca/applications/vnet_pci_dev.

1. Introduction

The DOCA VirtIO-net PCI Device Application emulates one or more virtio-net PCIe endpoints on NVIDIA BlueField DPUs. The application uses DOCA DevEmu PCI TLP APIs to emulate PCIe configuration/MMIO behavior and DOCA DevEmu VNet APIs to provide the hardware-backed VirtIO network datapath.

The host sees standard virtio-net PCI devices behind a software PCI switch topology. The DPU application owns PCI enumeration behavior, VirtIO PCI BAR emulation, MSI/MSI-X handling, hotplug signaling, queue lifecycle, control virtqueue handling, and optional live update handover.

1.1. System Design

1.1.1. Host Side (Virtio-net Client Driver)

The host runs the standard Linux virtio_net driver.

Host responsibilities include:

• Enumerating the emulated PCI switch and VirtIO-net endpoints.

• Negotiating VirtIO feature bits.

• Programming VirtIO common config registers.

• Allocating and configuring RX/TX virtqueues.

• Configuring MSI-X vectors.

• Sending queue notifications through the notify BAR.

• Sending control virtqueue commands, especially multi-queue commands such as VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

The host network interface appears as a normal Linux NIC and can be inspected with tools such as ip link, ethtool, and lspci.

1.1.2. BlueField DPU (DOCA Virtio-net Emulation Application)

The DPU runs doca vnet_pci_dev.

The application:

• Creates a DOCA VNet PCI TLP device type.

• Creates a software PCI topology with one upstream port, several downstream ports (one downstream port per endpoint), and one VirtIO-net endpoint per downstream port.

• Handles PCIe Config Read/Write and MMIO Read/Write TLPs.

• Emulates VirtIO PCI modern capabilities and network device configuration.

• Creates one DOCA VNet offload engine per endpoint.

• Creates RX/TX virtqueue objects and an optional control virtqueue.

• Uses a PCI config workqueue and worker progress engines to keep the main TLP progress loop responsive.

• Supports static endpoint creation and runtime hotplug/unplug.

• Supports active/standby live update through --lu-mode.

1.1.3. Data Path Provider

Unlike the vblk refapp, the vnet refapp does not expose a --provider DPA|DPU flag.

The packet datapath is provided by DOCA DevEmu VNet offload and the underlying NIC/firmware path after the VirtIO queues and offload engine are configured. The application itself is mostly a PCI/VirtIO control-plane and lifecycle orchestrator. It does not implement a userspace packet copy loop for RX/TX data queues.

1.2. End-to-End Flow

1. The DPU application starts and creates the PCI TLP type, TLP channel, software PCI topology, and VirtIO-net device model.

2. In static mode, endpoints are created immediately. In hotplug mode, endpoints are created after plug <idx>.

3. The host enumerates the PCI topology and discovers VirtIO-net endpoints.

4. The host virtio_net driver reads features and device configuration such as MAC, MTU, link status, speed, duplex, and maximum queue pairs.

5. The driver writes VirtIO status bits: ACKNOWLEDGE, DRIVER, FEATURES_OK, and DRIVER_OK.

6. On FEATURES_OK, the DPU prepares the offload engine, creates VQ objects, and creates the CVQ IO context if multi-queue is negotiated.

7. On DRIVER_OK, the DPU applies queue configuration, starts VQs, binds CVQ, and enables the offload engine.

8. Packet RX/TX proceeds through the DOCA VNet offload datapath.

9. Runtime CVQ commands can adjust the number of active queue pairs.

10. Reset, hotplug, unplug, shutdown, and live update paths tear down or transfer state while preserving PCI/VirtIO correctness.

2. Application Architecture

2.1. Core Components

Key files:

• vnet_pci_dev.c: main(), logging, signal handling, arguments registration, top-level config.

• vnet_pci_dev_core.c: application orchestration, topology setup, controller lifecycle, VirtIO status callback, hotplug, MQ handling, progress loop.

• vnet_pci_device.c: PCI config workqueue, TLP channel callback, config-space emulation, MMIO emulation, VirtIO PCI device model.

• vnet_pci_device.h: public interfaces, BAR layout, queue limits, struct vnet_pci_device.

• vnet_pci_dev_core.h: struct vnet_pci_dev_config, struct vnet_pci_dev_controller, feature defaults.

• pci_spec_tlp.h: PCI/TLP constants, topology macros, struct tlp_context, struct pci_device_config.

• vnet_virtio_types.h: VirtIO-net structures, status bits, feature bits, CVQ command definitions.

• vnet_pci_dev_lu.c and vnet_pci_dev_lu.h: live update active/standby handover.

Important data structures:

• struct tlp_context: global TLP/PCI state, DOCA device, PCI type, TLP channel, PE1, topology, BDF map, ACG queue, VirtIO devices, VNet controllers, and runtime parameters.

• struct vnet_pci_device: one per endpoint; holds VirtIO common config, network config, feature state, VQ shadow array, reset generation, and status callback.

• struct vnet_pci_dev_controller: one per endpoint; owns the DOCA VNet offload engine, RX/TX VQ arrays, CVQ, VNet IO context, worker PE, counters/stats, and atomic lifecycle state.

• PCI config workqueue: moves slow operations away from the main TLP progress loop.

• PE1: main progress engine for TLP handling.

• PE2: per-controller worker progress engines used by the workqueue and CVQ IO contexts.

2.1.1. Virtio-net Device Model

The app models a modern VirtIO-net PCI endpoint.

Each endpoint exposes:

• VirtIO PCI vendor/device IDs.

• PCI class code for network controller.

• PCIe capability.

• MSI-X capability.

• VirtIO common config capability.

• VirtIO notify capability.

• VirtIO ISR capability.

• VirtIO device config capability.

• VirtIO PCI config access capability.

The network device config contains:

• MAC address.

• Link status.

• Maximum virtqueue pairs.

• MTU.

• Speed.

• Duplex mode.

• RSS-related size/hash fields.

The base MAC comes from --mac-addr. Per-endpoint MACs are generated by incrementing the last octet by the endpoint index.

2.1.2. Virtio Queues

Queue model:

• RX VQ index: queue_pair_index * 2.

• TX VQ index: queue_pair_index * 2 + 1.

• CVQ index: max_queue_pairs * 2.

• Total VQs: max_queue_pairs * 2 + 1.

Defaults and limits:

• Default max queue pairs: 8.

• Maximum queue pairs: 127.

• Initial active queue pairs: 1.

• Default queue size: 1024.

• Queue size range: 16-4096.

• Queue size must be a power of two.

The application maintains a shadow copy of host-programmed VQ config in struct vnet_pci_device. Worker threads snapshot this shadow state before applying it to DOCA VQs, avoiding races with reset.

2.1.3. PCIe TLP Emulation

The application handles PCIe TLPs directly.

TLP handling flow:

1. vnet_pci_dev_event_cb() receives a TLP channel request.

2. ACG requests are cached for later MSI Memory Write TLPs.

3. PCI event requests are completed directly.

4. Normal requests are classified as config read/write, memory read/write, or unsupported.

5. The target device is resolved through BDF lookup or BAR address lookup.

6. Endpoint TLPs are protected with an endpoint read lock (to avoid racing hot-unplug destruction).

7. Config-space and MMIO handlers generate the correct completions.

8. Invalid/unmapped requests return Unsupported Request through the dummy device path.

The app emulates both Type 0 endpoint config space and Type 1 bridge config space.

2.1.4. PCI Switch Topology

The topology is:

USP
|-- DSP[0] -> EP[0] virtio-net
|-- DSP[1] -> EP[1] virtio-net
|-- ...
`-- DSP[N] -> EP[N] virtio-net

DUMMY device for unsupported BDF routing

Components:

• USP: upstream switch port, root of the software switch.

• DSP: downstream switch port, one per endpoint, with PCIe hotplug slot capability.

• EP: VirtIO-net endpoint.

• DUMMY: returns Unsupported Request for invalid probes.

The host assigns bus/device/function numbers during enumeration. The app caches BDF mappings and updates them on bus renumbering.

2.2. Initialization Process

The initialization process is:

1. Parse command line arguments.

2. Allocate and initialize tlp_context.

3. Open the DOCA device by IB device name or PCI address.

4. Create PE1 for TLP progress.

5. Configure optional CPU affinity.

6. Initialize the PCI config workqueue.

7. Verify VNet PCI TLP support.

8. Create the VNet PCI TLP type.

9. Attach the DOCA device to the PCI type.

10. Configure MSI-X count and doorbell count.

11. Start the PCI type.

12. Query and validate the predefined VirtIO-net BAR layout.

13. Create and configure the TLP channel.

14. Connect the TLP channel to PE1 and start it.

15. Initialize software PCI topology.

16. Initialize DOCA VNet subsystem.

17. Create VirtIO-net device models.

18. Create endpoints immediately in static mode, or defer endpoint creation in hotplug mode.

19. Allocate per-endpoint transaction regions.

20. Create one worker PE per controller.

21. Enter the progress loop.

2.3. Reset and Shutdown

Reset is triggered when the host writes device_status = 0.

The app:

• Marks the device as canceling in-progress worker operations.

• Increments the per-device reset generation.

• Reinitializes VirtIO queue shadow state.

• Holds device_status at NEEDS_RESET while controller cleanup is pending.

• Disables/stops/destroys VQs.

• Stops and destroys the CVQ IO context when needed.

• Keeps the offload engine object alive for normal driver rebind.

• Destroys the offload engine only during endpoint destruction or final shutdown.

Shutdown:

• Stops accepting new workqueue items.

• Waits for outstanding work and MQ-start threads.

• Destroys VirtIO device models.

• Destroys endpoint controllers, TLP devices, and representors.

• Flushes ACG credits.

• Stops and destroys the TLP channel.

• Stops/destroys PCI type and DOCA VNet subsystem.

• Destroys worker PEs and PE1.

• Frees tlp_context.

3. DOCA Libraries

The application uses:

• DOCA DevEmu PCI TLP: PCIe TLP channel, PCI type, TLP devices, ACG, MSI/MSI-X support.

• DOCA DevEmu VNet: VirtIO-net offload engine, RX/TX VQs, CVQ, counters.

• DOCA DevEmu VirtIO: generic VirtIO offload engine and VQ lifecycle APIs.

• DOCA PE: progress engines for TLP and worker-side asynchronous operations.

• DOCA Arg Parser: command-line parsing.

• DOCA Log: application and SDK logging.

• libibverbs: used by build and live update/device handling paths.

• Optional libbsd: used for strlcpy when available.

4. Dependencies

Required:

• BlueField platform with DOCA DevEmu VNet PCI TLP support.

• DOCA SDK including DevEmu support.

• libibverbs.

• Firmware/device support for VirtIO-net BAR layout.

• PCI switch/TLP emulation enabled in firmware.

The app checks that the VNet TLP type is supported and that the TLP channel exposes exactly one NV switch TLP DSP. If not, initialization fails and asks for firmware configuration adjustment.

Unlike the vblk refapp, this code does not create application-managed DMA buffer pools or hugepage-backed data buffers for packet data. Packet movement is handled by DOCA VNet offload.

5. Compiling the Application

5.1. Compiling All Applications

cd /opt/mellanox/doca/applications/
meson setup /tmp/build && ninja -C /tmp/build

5.2. Compiling Only the Current Application

cd /opt/mellanox/doca/applications/
meson setup /tmp/build -Denable_all_applications=false -Denable_vnet_pci_dev=true && ninja -C /tmp/build

Alternatively, edit applications/meson_options.txt:

• Set enable_all_applications to false.

• Set enable_vnet_pci_dev to true.

Then run the normal meson && ninja build.

5.3. Build Outputs

The build produces one executable:

doca_vnet_pci_dev

Typical output path:

/tmp/build/vnet_pci_dev/doca_vnet_pci_dev

Live update is implemented inside the same executable through --lu-mode; there are no separate *_lu and *_emu binaries in the current vnet app.

6. Running the Application

6.1. Prerequisites

A typical setup must enable TLP and PCI switch emulation in firmware. The code expects one NV switch TLP downstream port.

Example firmware configuration pattern:

mlxconfig -y -d /dev/mst/mt41692_pciconf0 reset
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_NUM_PF=32
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_ENABLE=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_ENABLE=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_TLP_PORT=1
mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_PORT=2

A cold reboot is required after firmware configuration changes.

6.2. Application Usage

Usage: doca_vnet_pci_dev [DOCA Flags] [Program Flags]

Common DOCA flags include:

• -h, --help

• -v, --version

• -l, --log-level

• --sdk-log-level

• --log-filter

• -j, --json

Program flags are described in the Command Line Flags section below.

6.2.1. Example Command Line Execution

Run with defaults in hotplug mode:

./doca_vnet_pci_dev

Run in static mode with four endpoints and 16 max queue pairs:

./doca_vnet_pci_dev -n 4 -q 16 -H 0

Run with custom MAC, MTU, and speed:

./doca_vnet_pci_dev -m 52:54:00:12:34:56 -t 9000 -s 100000

Run with explicit CPU affinity:

./doca_vnet_pci_dev --tlp-core 8 --worker-core 9 --mq-core 10

Run in live update active mode:

./doca_vnet_pci_dev --lu-mode active -H 0

Run a standby process for live update:

./doca_vnet_pci_dev --lu-mode standby

6.2.2. Runtime Commands

Available through stdin:

• plug <DSP_IDX>: create and hotplug an endpoint into a DSP slot. Hotplug mode only.

• unplug <DSP_IDX>: request hot-unplug of the endpoint in a DSP slot. Hotplug mode only.

• speed <EP_IDX> <Mbps>: update the reported link speed and raise a config-change MSI-X notification.

Examples:

plug 0
unplug 0
speed 0 25000

6.2.3. Verifying the Device (Host Side)

After startup or hotplug:

lspci | grep -i virtio
ip link

Inspect queue support:

ethtool -l <interface>

Change active queue pairs:

ethtool -L <interface> combined 4

Inspect link information:

ethtool <interface>

6.3. Command Line Flags

6.3.1. General Flags

The app uses DOCA Arg Parser, so standard DOCA flags are available:

• -h, --help: print usage.

• -v, --version: print version.

• -l, --log-level: set application log level.

• --sdk-log-level: set SDK log level.

• --log-filter: filter logs by module.

• -j, --json: parse flags from JSON.

6.3.2. Program Flags

• -p, --pci-addr: DOCA device PCI address. Default: 0000:03:00.0.

• -d, --ibdev-name: IB device name. Overrides --pci-addr.

• -m, --mac-addr: base MAC address. Default: 52:54:00:12:34:56.

• -t, --mtu: MTU, range 68-9000. Default: 1500.

• -s, --speed: link speed in Mbps. Default: 100000.

• -D, --duplex: duplex mode, 0 half or 1 full. Default: 1.

• -q, --max-queue-pairs: maximum queue pairs, range 1-127. Default: 8.

• -z, --queue-size: queue size, power of two, range 16-4096. Default: 1024.

• -n, --num-ep: number of endpoints, range 1-32. Default: 1.

• -H, --hotplug-mode: 1 hotplug, 0 static. Default: 1.

• --tlp-core: CPU core for the main TLP progress thread, or -1 for auto.

• --worker-core: CPU core for the PCI config worker thread, or -1 for auto.

• --mq-core: CPU core for MQ start threads, or -1 for auto.

• --lu-mode: live update mode: none, active, or standby. Default: none.

Validation rules:

• --tlp-core must not equal --worker-core when both are explicitly set.

• --mq-core must not equal --tlp-core when both are explicitly set.

• --queue-size must be a power of two.

• --mac-addr last octet must leave enough room for per-endpoint incrementing.

6.4. Troubleshooting

Common issues:

• VNet TLP type unsupported: the device or firmware does not expose VirtIO-net emulation support.

• Wrong TLP port count: the app expects exactly one NV switch TLP DSP.

• Host does not see devices: verify firmware settings, run PCI rescan, and confirm the app is in static mode or that plug <idx> was issued.

• Hotplug command rejected: host hotplug interrupt may not be enabled yet, or a previous unplug is still in progress.

• Queue startup takes time with high --max-queue-pairs: MQ expansion is intentionally deferred and backgrounded.

• RSS/hash CVQ commands fail: the app does not implement those CVQ commands.

• Host stalls during development changes: keep slow DOCA operations off PE1; the existing workqueue design is there to avoid PCIe completion timeouts.

7. Application Code Flow

7.1. Framework Initialization

7.1.1. Initialization Flow

1. main() initializes logging.

2. doca_argp_init() initializes argument parsing.

3. register_vnet_pci_dev_params() registers application-specific flags.

4. doca_argp_start() parses CLI/JSON input.

5. Signal handlers are installed for SIGINT and SIGTERM.

6. vnet_pci_dev_run() receives the parsed config.

7. init_tlp_context() allocates topology state.

8. find_doca_device() opens the DOCA device by IB name or PCI address.

9. init_progress_engine() creates PE1.

10. vnet_pci_device_configure_affinity() records affinity preferences.

11. init_virtio_network_device() initializes PCI/TLP, topology, VNet, VirtIO device state, and endpoint creation.

12. Worker PEs are created and attached to the PCI config workqueue.

13. run_progress_loop() starts processing TLPs, runtime commands, MSI retries, and LU triggers.

7.1.2. Key Points

• PE1 is reserved for TLP handling and must remain responsive.

• Heavy operations are submitted to the PCI config workqueue.

• Worker PEs are created per controller so multi-endpoint operations can progress independently.

• In LU standby mode, device and channel restore alter the normal open/start flow.

7.2. Resource Management

7.2.1. Resource Management Flow

Main resources:

• tlp_context

• DOCA device handle

• PE1 main progress engine

• VNet PCI TLP type

• TLP channel

• ACG credit queue

• software PCI device configs

• BDF map

• VirtIO device models

• VNet controller array

• per-controller worker PEs

• per-controller offload engine

• RX/TX/CVQ objects

• VNet IO context for CVQ

• stats/counters handles

• per-endpoint transaction region memory

• PCI config workqueue and work-item pool

Resource creation order:

1. Allocate tlp_context.

2. Initialize locks, maps, endpoint arrays, and controller arrays.

3. Open DOCA device.

4. Create PE1.

5. Initialize PCI config workqueue.

6. Create/start PCI type.

7. Create TLP channel.

8. Initialize topology and VNet subsystem.

9. Create VirtIO device models.

10. Create endpoints if static mode.

11. Allocate transaction regions.

12. Create worker PEs.

13. Create offload/VQ/IO resources lazily as the host reaches VirtIO states.

7.2.2. Key Points

• The workqueue uses a preallocated pool of 1024 work items to avoid hot-path allocation.

• MQ start concurrency is limited to 2 to avoid memory pressure and firmware command saturation.

• Reset generation fields prevent stale async work from affecting a post-reset device.

• Endpoint read/write locks protect TLP handlers from hotplug destruction races.

• Diagnostics run on the worker side, not PE1.

7.3. PCI Switch Topology and Device Configuration

7.3.1. PCI Switch Topology Flow

1. init_device_topology() initializes all bridge and endpoint config structures.

2. USP is configured as a Type 1 bridge with upstream-port PCIe capability.

3. Each DSP is configured as a Type 1 bridge with downstream-port and hotplug slot capabilities.

4. Each endpoint is configured as a Type 0 network device.

5. The dummy device is configured for unsupported request handling.

6. In static mode, create_all_devices() creates endpoint representors and TLP devices.

7. In hotplug mode, create_device() is called later from the hotplug work item.

8. Host enumeration assigns bus numbers and BDFs.

9. BDF mappings are cached and updated if the host renumbers buses.

Endpoint creation flow:

1. Create or reattach a representor.

2. Query representor VHCA ID.

3. Create the TLP device.

4. Start the TLP device.

5. Create the VNet controller and offload engine.

6. Mark endpoint present.

7.3.2. Key Points

• DSP count equals endpoint count.

• Each DSP supports PCIe hotplug slot behavior.

• Hotplug signaling uses attention button and slot status bits.

• MSI for hotplug is sent as a Memory Write TLP using ACG credits.

• If ACG credits are exhausted, MSI retry is scheduled.

• Static mode sets slots to powered-on and link-active at startup.

7.4. Virtio-net Controller

7.4.1. Controller Initialization Flow

Controller creation happens in vnet_pci_dev_vnet_controller_create():

1. Resolve endpoint PF index.

2. Associate controller with the per-endpoint VirtIO device.

3. Convert TLP device to PCI endpoint.

4. Create a VNet offload engine, or import one during LU standby.

5. Configure MTU.

6. Configure MAC address.

7. Configure total number of queues.

8. Initialize controller atomics and deferred MQ state.

9. Allocate RX and TX VQ pointer arrays.

10. Query/log SF representor information.

11. Wait for host FEATURES_OK before starting hardware preparation.

On FEATURES_OK:

1. Determine if VIRTIO_NET_F_CTRL_VQ and VIRTIO_NET_F_MQ were negotiated.

2. Set initial active queue pairs.

3. Submit offload engine start.

4. Submit VQ object creation.

5. Submit CVQ IO context creation.

On DRIVER_OK:

1. Submit start-and-enable work.

2. Snapshot VQ shadow config.

3. Configure RX/TX/CVQ VQs.

4. Start VQs.

5. Bind CVQ to IO context.

6. Enable VirtIO offload engine.

7. Create stats/counter resources.

7.4.2. Virtio Queue Lifecycle State Machine

Conceptual state flow:

UNCREATED -> CREATED -> CONFIGURED -> STARTED -> ENABLED
     ^                                            |
     |                                            v
DESTROYED <- DESTROYING <- STOPPED <- DISABLED <- RESET

Lifecycle by phase:

• UNCREATED: no DOCA VQ object exists.

• CREATED: RX/TX/CVQ object has been created after FEATURES_OK.

• CONFIGURED: host-provided queue size, MSI-X vector, descriptor address, driver address, and device address have been applied.

• STARTED: doca_devemu_virtio_vq_start() has completed.

• ENABLED: offload engine or VQ group enable has made the queue operational.

• DISABLED: VQ group or engine disable has stopped host-visible processing.

• STOPPED: VQ stop completed.

• DESTROYED: VQ object destroyed.

7.4.3. Key Points

• RX/TX VQs are created for the configured maximum queue pairs.

• CVQ is created only when MQ is negotiated.

• Data queues may be deferred during multi-endpoint startup so enumeration can settle.

• Increasing queue pairs completes the CVQ request immediately, then starts extra queues in background.

• Decreasing queue pairs disables and stops extra queues synchronously.

• Worker readers use snapshots of VQ shadow config to avoid reset races.

7.5. I/O Request Processing

7.5.1. I/O Request Processing Flow

For packet data:

1. Host virtio_net driver posts RX buffers and TX descriptors.

2. Host notifies queues through the VirtIO notify region.

3. DOCA VNet offload processes packet datapath using the configured VQs.

4. The application does not copy packet payloads in userspace.

5. Completion and interrupt behavior are handled through the DOCA VNet/VirtIO offload path.

For CVQ control requests:

1. Host submits a control command on CVQ.

2. DOCA VNet forwards the request to the registered IO context.

3. Worker PE progresses the IO context.

4. vnet_pci_dev_ctrl_req_handler() receives cls and cmd.

5. MQ pair-set commands are handled by the app.

6. MAC/RX/VLAN classes are acknowledged as backend-handled.

7. Unsupported classes/commands return VIRTIO_NET_ERR.

8. The request is completed with doca_devemu_vnet_ctrl_req_complete().

7.5.2. Key Points

• RX/TX data path is hardware/offload driven.

• CVQ is the main application-visible I/O request path.

• MQ expansion avoids blocking the host by completing the CVQ command before long VQ startup.

• RSS/hash CVQ commands are not implemented by the app.

• The app’s most important responsibility is correct lifecycle sequencing, not packet processing.

7.6. Device Lifecycle Management

7.6.1. Device Lifecycle State Machine

Endpoint-level conceptual state:

INIT
  -> MODEL_CREATED
  -> ENDPOINT_CREATED
  -> FEATURES_OK_PREP
  -> DRIVER_OK_ENABLE
  -> OPERATIONAL
  -> RESETTING
  -> CLEANED_FOR_REBIND
  -> DESTROYED

Hotplug state:

ABSENT -> PLUG_REQUESTED -> PRESENT_LINK_DOWN -> POWERED_ON -> OPERATIONAL
   ^                                                              |
   |                                                              v
DESTROYED  <-   POWERED_OFF  <-   UNPLUG_PENDING  <-   UNPLUG_REQUESTED

7.6.2. Hotplug Management Flow

Plug:

1. Runtime command plug <idx> is parsed.

2. Workqueue receives hotplug work.

3. The endpoint representor, TLP device, and VNet controller are created.

4. DSP slot status sets attention button, presence detect changed, and presence detect state.

5. MSI is sent if ACG credit and host MSI setup are available.

6. Host pciehp notices the slot event.

7. Host powers on the slot.

8. TLP handler observes Slot Control Power ON and sets DLActive.

9. Host enumerates and probes the VirtIO-net endpoint.

Unplug:

1. Runtime command unplug <idx> is parsed.

2. Workqueue marks endpoint pending_unplug.

3. DSP slot status raises attention button.

4. MSI is sent or retried.

5. Host pciehp unbinds the driver and powers off the slot.

6. TLP handler observes Slot Control Power OFF.

7. Delayed destroy work is submitted.

8. Controller, TLP device, and representor are destroyed.

9. Pending flags are cleared.

If the host never powers off the slot, the main loop forces removal after UNPLUG_TIMEOUT_SEC, currently 30 seconds.

7.6.3. Key Points

• Hotplug creation is intentionally done on the workqueue because endpoint creation can take hundreds of milliseconds.

• PE1 must keep progressing TLPs during hotplug.

• Plug and unplug use PCIe hotplug slot semantics rather than ad hoc host signaling.

• Unplug waits for host power-off before destroying resources.

• A timeout safety net handles failed or stuck host removal.

7.7. Cleanup and Shutdown

7.7.1. Cleanup and Shutdown Flow

Shutdown begins when SIGINT or SIGTERM sets force_quit.

The main progress loop exits when:

• Quit is requested and no hot-unplug is pending.

• Hot-unplug pending has completed.

• Hot-unplug pending times out.

• LU handover is requested by SIGUSR1.

7.7.2. Phase 1 – Worker and Device Cleanup

The current vnet app does not have vblk-style per-IO worker threads named like vblk_io_ctx_thread(). Its equivalent first phase is workqueue and controller-side cleanup.

Steps:

1. Disable worker-side diagnostics.

2. Shut down PCI config workqueue.

3. Join outstanding MQ start threads.

4. Stop accepting new work.

5. Destroy VirtIO device models.

6. For each endpoint, destroy controller resources.

7. Disable VQs while engine is enabled.

8. Disable engine.

9. Stop VQs.

10. Destroy VQs.

11. Stop and destroy CVQ IO context.

12. Stop offload engine.

13. Destroy offload engine when required.

14. Destroy TLP devices and representors.

15. Clear endpoint present state.

7.7.3. Phase 2 – Reverse-Order Resource Cleanup

After workqueue shutdown:

1. Clear worker PE references from the workqueue.

2. Destroy per-controller worker PEs.

3. Flush pending ACG credits.

4. Stop the TLP channel context.

5. Drain PE1 until the TLP channel reaches idle or timeout.

6. Destroy the TLP channel.

7. Stop and destroy the PCI type.

8. Tear down DOCA VNet.

9. Remove the DOCA device from VNet.

10. Destroy PE1.

11. Free transaction regions, BDF maps, topology arrays, controller arrays, and VirtIO device arrays.

12. Close LU connections if any.

13. Destroy ARGP resources in main().

7.7.4. Key Points

• Cleanup order follows ownership dependencies.

• Workqueue shutdown happens before freeing objects referenced by work items.

• Worker PEs are destroyed only after the workqueue exits.

• TLP channel is stopped and drained before destruction.

• LU cleanup differs because the active process transfers traffic ownership before releasing local handles.

7.8. Live Upgrade (LU) Mode

The vnet refapp implements live update in the same binary through --lu-mode.

Modes:

• none: normal operation.

• active: current traffic owner.

• standby: process that restores state from active and takes over.

LU uses:

• Unix domain socket: /tmp/vnet_live_update.sock.

• Lock file: /tmp/vnet_live_update.lock.

• Shared memory: /vnet_live_update.

• TLP channel shared memory directory: /dev/shm/vnet_lu_channel.

• SIGUSR1 to trigger active handover.

High-level LU flow:

1. Active process runs and serves traffic.

2. Standby process connects to active.

3. Active sends command FD and exports device state.

4. Standby opens shared memory and reconstructs DOCA device state.

5. Standby recreates endpoint/controller/offload objects from export blobs.

6. Standby replays VirtIO config and VQ state without enabling engines.

7. Standby signals DEV_READY.

8. Active sends final DEV_GO.

9. Standby enables engines, one or more in parallel.

10. Standby sends DEV_ACK.

11. Channel LU transfers the TLP channel.

12. Standby becomes active and can support another chained LU.

LU state includes:

• endpoint VHCA ID

• MAC/MTU

• device features

• max queue pairs

• active queue pairs

• queue size

• MQ negotiation state

• per-VQ descriptor/driver/device addresses

• MSI-X vectors

• offload engine export blobs

• TLP channel configuration

7.9. Crash Recovery

The current vnet refapp has live update state transfer but does not implement a separate persistent crash recovery mechanism comparable to a full crash-resume feature.

Important distinction:

• LU is coordinated and stateful: active and standby cooperate.

• Crash recovery would require reconstructing after unexpected termination without a live handover.

• The current code has robust cleanup, reset handling, timeout handling, and LU import/export paths, but no standalone crash recovery workflow is exposed as an application mode.

8. Virtio-net Operations Support Summary

Supported by the refapp/offload stack:

• RX/TX packet queues: supported through DOCA DevEmu VNet offload after VQs are configured and engine is enabled.

• Control virtqueue: supported when VIRTIO_NET_F_CTRL_VQ and VIRTIO_NET_F_MQ are negotiated.

• Multi-queue pair set: implemented for VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

• Runtime link speed update: supported through app command speed <idx> <Mbps> and config-change MSI-X.

• Hotplug/unplug: supported through PCIe hotplug slot emulation.

Partially supported or backend-acknowledged:

• MAC control class: acknowledged as backend-handled in the app callback.

• RX control class: acknowledged as backend-handled in the app callback.

• VLAN control class: acknowledged as backend-handled in the app callback.

Not implemented in the app callback:

• RSS configuration command.

• Hash configuration command.

• Unknown MQ commands.

• Unknown control classes.

9. Virtio Feature Bits Support Summary

Advertised by default:

• VIRTIO_NET_F_MAC: device reports MAC address.

• VIRTIO_NET_F_STATUS: link status is available.

• VIRTIO_NET_F_CSUM: checksum offload feature advertised.

• VIRTIO_NET_F_MTU: MTU is reported.

• VIRTIO_NET_F_SPEED_DUPLEX: speed and duplex are reported.

• VIRTIO_NET_F_HOST_TSO4: host can send TSOv4 packets to device.

• VIRTIO_NET_F_HOST_TSO6: host can send TSOv6 packets to device.

Always added by device initialization:

• VIRTIO_F_VERSION_1: modern VirtIO compliance.

• VIRTIO_F_ACCESS_PLATFORM: platform/IOMMU access semantics.

• VIRTIO_NET_F_CTRL_VQ: control virtqueue available.

• VIRTIO_NET_F_MQ: multi-queue support.

Defined but not part of the default advertised feature set unless code (SDK) is extended:

• VIRTIO_NET_F_GUEST_CSUM

• VIRTIO_NET_F_CTRL_GUEST_OFFLOADS

• VIRTIO_NET_F_GUEST_TSO4

• VIRTIO_NET_F_GUEST_TSO6

• VIRTIO_NET_F_GUEST_ECN

• VIRTIO_NET_F_GUEST_UFO

• VIRTIO_NET_F_HOST_ECN

• VIRTIO_NET_F_HOST_UFO

• VIRTIO_NET_F_MRG_RXBUF

• VIRTIO_NET_F_CTRL_RX

• VIRTIO_NET_F_CTRL_VLAN

10. Developer Notes

The main engineering theme in this refapp is latency isolation. PE1 must continuously progress PCI TLP completions, so any operation that can block on firmware or DOCA state transitions is moved to the PCI config workqueue or detached MQ-start threads.

The second key theme is reset safety. The app uses reset generations, cancellation flags, VQ shadow snapshots, and short bounded waits so stale worker operations cannot corrupt a newly reset device.

The third key theme is host compatibility. Hotplug follows PCIe slot semantics, VirtIO status transitions follow the expected driver lifecycle, and CVQ MQ commands are completed quickly to avoid Linux probe and soft-lockup issues.

Last updated: June 02, 2026