1.1. Introduction
The DOCA DevEmu Virtio-net library provides low-level APIs for building emulated VirtIO network devices on NVIDIA BlueField DPUs. It is part of the DOCA DevEmu Virtio subsystem and is intended to be used with DOCA DevEmu PCI TLP emulation.
The reference application, `vnet_pci_dev`, demonstrates how to build a VirtIO-net PCI device controller around this library. The application owns PCI/TLP emulation, VirtIO feature negotiation, virtqueue lifecycle, control virtqueue handling, hotplug, reset, and live-update coordination. Packet data-path processing is not surfaced as per-packet application callbacks; RX/TX virt-queues are handled by the vNET offload library and hardware path once the offload engine is enabled.
1.1.1. Architecture and Library Interaction
The DevEmu Virtio-net stack follows the same layered design as other DOCA DevEmu Virtio libraries:
DOCA DevEmu PCI: manages the TLP PCI transport: PCI type creation, representors, endpoints, TLP channel events, MSI-X, and hotplug support.
DOCA DevEmu Virtio Common: manages common VirtIO offload concepts: offload engine lifecycle, virtqueue configuration, virtqueue start/stop/enable/disable, IO binding, virtqueue debug state, and reset-sensitive state transitions.
DOCA DevEmu Virtio-net: manages VirtIO-net-specific objects: network offload engine, RX/TX/control virtqueues, MAC/MTU configuration, representor lookup, control virtqueue request routing, and DevEmu Virtio-net counters.
The reference app adds a complete VirtIO PCI device model around the library. It implements config-space and MMIO TLP handling, tracks VirtIO common configuration, exposes network device configuration, and drives the library according to host driver status transitions.
1.1.2. Core Functionality
The DevEmu Virtio-net library provides these main capabilities:
- Create a VirtIO-net PCI TLP type.
- Create a per-endpoint VirtIO-net offload engine.
- Configure network-specific parameters such as MAC address and MTU.
- Create RX, TX, and control virtqueues.
- Start, enable, disable, stop, and destroy virtqueues through the common VirtIO API.
- Route control virtqueue commands to the application as `doca_devemu_vnet_ctrl_req`.
- Expose DevEmu Virtio-net virtqueue counters and common VirtIO queue debug state.
- Export/import offload engine state for live update and recovery flows.
A key difference from DevEmu Virtio-blk: DevEmu Virtio-net does not expose per-packet request objects like doca_devemu_vblk_req . The application does not receive every packet. Instead, it configures RX/TX virtqueues and lets the DevEmu Virtio-net offload engine handle packet movement. The only request-style event exposed to the app is the control virtqueue request.
1.1.3. User Responsibilities
A DevEmu Virtio-net controller application is responsible for:
- Creating and configuring the PCI TLP type.
- Creating PCI representors and TLP endpoint devices.
- Emulating VirtIO PCI configuration and device-specific network config.
- Advertising feature bits consistently with what the application and library can support.
- Creating, configuring, starting, binding, enabling, disabling, stopping, and destroying RX/TX/CVQ virtqueues.
- Creating an IO context only when control virtqueue commands need to be routed to the app.
- Handling doca_devemu_vnet_ctrl_req events and completing them with VirtIO-net ACK status.
- Handling device reset, hotplug, and teardown without blocking the TLP progress path.
- Managing any backend network policy, representor setup, or system integration outside the DevEmu Virtio-net library.
1.2. Environment and Prerequisites
VirtIO-net TLP emulation depends on BlueField TLP device emulation support. The environment should be configured for DPU mode and TLP emulation.
-
Configure BlueField to work in DPU mode as described in BlueField Modes of Operation.
-
Enable TLP emulation by running the following on the DPU:
[dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_ENABLE=1 [dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_NUM_PF=32 -
Perform a BlueField system cold reboot for the
mlxconfigsettings to take effect.
For hotplug, PCI switch emulation and host kernel support are also required.
-
for Host platforms with intel CPU, need config grub file with:
intel_iommu=on iommu=pt pci=realloc -
for DPU side, need config with:
[dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_ENABLE=1 [dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_TLP_PORT=1 [dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_PORT=2PCI_SWITCH_EMULATION_NUM_PORTequals 1 plus the number of hot-plugged PCIe functions.
for 32 vNET devices, need configmlxconfig -y -d /dev/mst/mt41692_pciconf0 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_SF_BAR_SIZE=8 PF_TOTAL_SF=33.
The vnet_pci_dev reference app additionally expects exactly one physical NV switch TLP downstream port for its topology.
During initialization it queries the TLP channel and fails if the number of downstream ports is not `1`.
The reference topology is:
USP
|-- DSP[0] -> EP[0] virtio-net
|-- DSP[1] -> EP[1] virtio-net
|-- ...
`-- DSP[N] -> EP[N] virtio-net
- One upstream bridge, or USP.
- `N` downstream bridge slots, or DSPs.
- `N` emulated VirtIO-net endpoints.
1.3. Main Abstractions
1.3.1. Virtio-net Type
The VirtIO-net type is a PCI TLP type specialized for VirtIO-net emulation.
Important APIs:
- `doca_devemu_vnet_cap_is_pci_tlp_type_supported()`
- `doca_devemu_vnet_pci_tlp_type_create()`
- `doca_devemu_pci_type_set_dev()`
- `doca_devemu_pci_type_set_num_msix()`
- `doca_devemu_pci_type_set_num_db()`
- `doca_devemu_pci_tlp_type_set_pci_cap_conf()`
- `doca_devemu_pci_type_start()`
The reference app creates the type in `vnet_pci_device_init()`, sets MSI-X and doorbell counts, configures MSI-X capability placement, starts the type, and then queries the predefined VirtIO-net BAR layout provided by firmware.
1.3.2. Virtio-net Offload Engine
The offload engine is the per-endpoint DevEmu Virtio-net object that owns virtqueue state and network-specific configuration.
Important APIs:
- `doca_devemu_vnet_offload_engine_create()`
- `doca_devemu_vnet_offload_engine_create_from_export()`
- `doca_devemu_vnet_offload_engine_export()`
- `doca_devemu_vnet_offload_engine_destroy()`
- `doca_devemu_vnet_offload_engine_set_mtu()`
- `doca_devemu_vnet_offload_engine_set_mac()`
- `doca_devemu_vnet_offload_engine_get_rep()`
- `doca_devemu_vnet_offload_engine_as_virtio_offload()`
After casting to common VirtIO offload:
- `doca_devemu_virtio_offload_engine_set_num_queues()`
- `doca_devemu_virtio_offload_engine_start()`
- `doca_devemu_virtio_offload_engine_enable()`
- `doca_devemu_virtio_offload_engine_disable()`
- `doca_devemu_virtio_offload_engine_stop()`
The reference app creates the offload engine when a PCI endpoint is created, sets MTU and MAC, configures total queue count, but deliberately delays `doca_devemu_virtio_offload_engine_start()` until the host reaches `FEATURES_OK`.
1.3.3. Virtqueues
VirtIO-net uses separate queue types:
- RX virtqueue: `doca_devemu_vnet_rx_vq`
- TX virtqueue: `doca_devemu_vnet_tx_vq`
- Control virtqueue: `doca_devemu_vnet_ctrl_vq`
Key APIs:
- `doca_devemu_vnet_rx_vq_create()`
- `doca_devemu_vnet_tx_vq_create()`
- `doca_devemu_vnet_ctrl_vq_create()`
- `doca_devemu_vnet_rx_vq_as_vq()`
- `doca_devemu_vnet_tx_vq_as_vq()`
- `doca_devemu_vnet_ctrl_vq_as_vq()`
- `doca_devemu_vnet_rx_vq_destroy()`
- `doca_devemu_vnet_tx_vq_destroy()`
- `doca_devemu_vnet_ctrl_vq_destroy()`
The reference app uses queue index layout:
- RX queue for queue pair `i`: `2 * i`
- TX queue for queue pair `i`: `2 * i + 1`
- Control queue: `max_queue_pairs * 2`
- Total queues: `max_queue_pairs * 2 + 1`
The control queue is created only when both `VIRTIO_NET_F_CTRL_VQ` and `VIRTIO_NET_F_MQ` are negotiated.
1.3.4. Virtio-net IO Context
The DevEmu Virtio-net IO context routes control virtqueue requests to the application. It is not a packet datapath callback context.
Key APIs:
- `doca_devemu_vnet_io_create_from_offload_engine()`
- `doca_devemu_vnet_io_destroy()`
- `doca_devemu_vnet_io_as_ctx()`
- `doca_devemu_vnet_io_as_virtio_io()`
- `doca_devemu_vnet_io_event_vnet_ctrl_req_register()`
The reference app creates the IO context only when MQ/CVQ is active. It connects the IO context to a worker progress engine, registers `vnet_pci_dev_ctrl_req_handler()`, starts the context, and later binds the CVQ with `doca_devemu_virtio_io_bind_vq()`.
1.3.5. Control Request
A control request represents one VirtIO-net CVQ command.
Handler signature:
- void handler(struct doca_devemu_vnet_ctrl_req *req, uint8_t cls, uint8_t cmd, void *req_user_data);
key APIs:
- `doca_devemu_vnet_ctrl_req_get_vnet_io()`
- `doca_devemu_vnet_ctrl_req_get_vq_user_data()`
- `doca_devemu_vnet_ctrl_req_get_data()`
- `doca_devemu_vnet_ctrl_req_get_data_len()`
- `doca_devemu_vnet_ctrl_req_get_result()`
- `doca_devemu_vnet_ctrl_req_get_result_len()`
- `doca_devemu_vnet_ctrl_req_complete()`
The reference app handles `VIRTIO_NET_CTRL_MQ / VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET`. MAC, RX, and VLAN control classes are acknowledged as backend-handled in the sample.
1.4. Feature Bits
The reference app advertises these default network features:
- `VIRTIO_NET_F_CSUM`
- `VIRTIO_NET_F_MTU`
- `VIRTIO_NET_F_MAC`
- `VIRTIO_NET_F_STATUS`
- `VIRTIO_NET_F_HOST_TSO4`
- `VIRTIO_NET_F_HOST_TSO6`
- `VIRTIO_NET_F_SPEED_DUPLEX`
It always adds:
- `VIRTIO_F_VERSION_1`
- `VIRTIO_F_ACCESS_PLATFORM`
- `VIRTIO_NET_F_CTRL_VQ`
- `VIRTIO_NET_F_MQ`
Even though the app advertises control queue and multiqueue support, it still checks what the driver negotiated. If both `CTRL_VQ` and `MQ` are not negotiated, the app skips CVQ creation and IO context initialization and runs with one active queue pair.
1.5. Device Configuration Layout
The reference app models the VirtIO-net device config as:
struct vnet_virtio_net_config {
uint8_t mac[ETH_ALEN];
uint16_t status;
uint16_t max_virtqueue_pairs;
uint16_t mtu;
uint32_t speed
...
};
The app initializes this from CLI/config values:
- MAC address
- Link status
- Maximum queue pairs
- MTU
- Speed
The common VirtIO config and queue shadow state are maintained by the application, not by the vNET library. Host writes to queue addresses, queue size, queue MSI-X vector, queue enable, device status, and feature selectors are decoded from TLP MMIO/config accesses and reflected into the app’s shadow structures.
1.6. Global Configuration
The current DevEmu Virtio-net public API has a smaller global configuration surface than DevEmu Virtio-blk.
Key APIs:
- `doca_devemu_vnet_add_dev()`
- `doca_devemu_vnet_rm_dev()`
- `doca_devemu_vnet_set_vnet_ctrl_req_user_data_size()`
- `doca_devemu_vnet_get_vnet_ctrl_req_user_data_size()`
- `doca_devemu_vnet_init()`
- `doca_devemu_vnet_teardown()`
Unlike DevEmu Virtio-blk, the DevEmu Virtio-net headers do not expose global `seg_max`, queue-size, or datapath-on-DPA configuration APIs.
The reference app performs:
1. Add DOCA device to DevEmu Virtio-net.
2. Initialize the DevEmu Virtio-net subsystem.
3. Parse and store MAC/MTU/queue-pair settings.
4. Create per-endpoint offload engines later.
1.7. Discovery and PCI Setup
A typical DevEmu Virtio-net TLP setup flow is:
1. Open a DOCA device by PCI address or IB device name.
2. Query `doca_devemu_vnet_cap_is_pci_tlp_type_supported()`.
3. Create the DevEmu Virtio-net PCI TLP type with `doca_devemu_vnet_pci_tlp_type_create()`.
4. Attach the DOCA device to the PCI type.
5. Configure MSI-X and doorbell counts.
6. Configure PCI capabilities such as MSI-X.
7. Start the PCI type.
8. Create a TLP channel with `doca_devemu_pci_tlp_channel_create()`.
9. Register the TLP request handler.
10. Connect the TLP channel context to the main progress engine.
11. Start the TLP channel context.
12. Create endpoint representors and TLP devices.
The reference app uses a main progress engine for TLP handling and one worker progress engine per controller for slower DOCA operations.
1.8. Initialization Flow
1.8.1. Application Startup
At process startup, the app:
1. Allocates `tlp_context`.
2. Opens the DOCA device.
3. Creates the main progress engine.
4. Initializes the DevEmu Virtio-net PCI framework and workqueue.
5. Creates and starts the vNET PCI TLP type.
6. Creates and starts the TLP channel.
7. Connects the TLP channel to the main PE.
8. Initializes software topology.
9. Calls DevEmu Virtio-net global init.
10. Creates the software VirtIO-net device model.
11. In static mode, creates all endpoint representors and TLP devices.
12. Creates one DevEmu Virtio-net controller per endpoint.
13. Creates one worker PE per controller.
14. Enters the progress loop.
1.8.2. Per-endpoint Controller Creation
For each endpoint, the app:
1. Creates or reattaches a PCI representor.
2. Creates a TLP device.
3. Starts the TLP device.
4. Creates a vNET offload engine from the PCI endpoint.
5. Sets MTU.
6. Sets MAC address.
7. Sets total number of queues on the common VirtIO offload engine.
8. Allocates RX/TX VQ pointer arrays.
9. Queries the associated representor for logging/debug.
The offload engine is created before host driver probing, but not started until `FEATURES_OK`.
1.8.3. FEATURES_OK
When the host writes `FEATURES_OK`, the app queues asynchronous work:
1. Determine whether `CTRL_VQ` and `MQ` were both negotiated.
2. Set `num_active_qps` to `1`.
3. Set `max_queue_pairs` to the configured maximum if MQ is active, otherwise `1`.
4. Start the offload engine.
5. Create RX/TX VQs for all configured queue pairs.
6. Create CVQ only if MQ/CVQ was negotiated.
7. Create and start the IO context only if CVQ is needed.
This work is queued to avoid blocking TLP completions on slow DOCA or firmware operations.
1.8.4. DRIVER_OK
When the host writes `DRIVER_OK`, the app:
1. Verifies the first queue pair is fully configured.
2. Applies queue configuration with `doca_devemu_virtio_vq_set_conf()`.
3. Starts active RX/TX VQs with `doca_devemu_virtio_vq_start()`.
4. Starts the CVQ if configured.
5. Binds the CVQ to the IO context.
6. Enables the offload engine with `doca_devemu_virtio_offload_engine_enable()`.
7. Creates DevEmu Virtio-net counters for debug/statistics.
At this point the device is operational.
1.9. Execution Phase
1.9.1. Control Path
The control path includes:
- TLP request handling.
- PCI config and MMIO emulation.
- VirtIO feature negotiation.
- Device status transitions.
- Queue configuration writes.
- Queue lifecycle.
- CVQ command handling.
- Config-change MSI-X notification.
- Reset and hotplug.
The app keeps TLP handling latency-sensitive. Expensive work is moved to a workqueue and worker PEs. This is essential because slow endpoint creation or large multiqueue startup can otherwise cause PCIe completion timeouts or host stalls.
1.9.2. Data Path
The data path is the most important conceptual difference from DevEmu Virtio-blk.
For DevEmu Virtio-net, the app does not receive every packet. RX/TX virtqueues are configured and then owned by the DevEmu Virtio-net offload engine. Once queues are started and the engine is enabled, packet movement is handled by the DevEmu Virtio-net offload path.
The application still owns:
- Virtqueue creation.
- Virtqueue configuration from host-provided ring addresses.
- Virtqueue start/stop/enable/disable.
- Multiqueue control.
- Debug counters and virtqueue stats.
- Representor and integration logic.
The app handles CVQ commands through `doca_devemu_vnet_ctrl_req`, not packet payloads.
1.9.3. Control Virtqueue Path
The CVQ path is:
1. Host submits a VirtIO-net control command.
2. DevEmu Virtio-net library receives the CVQ command.
3. The registered IO handler is invoked with `req`, `cls`, and `cmd`.
4. The app reads command-specific data using `doca_devemu_vnet_ctrl_req_get_data()`.
5. The app optionally writes command-specific result data using `doca_devemu_vnet_ctrl_req_get_result()`.
6. The app completes with `doca_devemu_vnet_ctrl_req_complete(req, ack, len)`.
For `VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET`, the app reads the requested queue-pair count and validates it.
If the request decreases active queue pairs, extra queues are disabled/stopped synchronously.
If the request increases active queue pairs, the app completes the CVQ request immediately with `VIRTIO_NET_OK` and starts additional queues asynchronously. This avoids blocking the host in `virtnet_send_command()` while many queues are started.
1.10. Reset Flow
When the host writes `device_status = 0`, the app treats it as a VirtIO device reset.
The reset path:
1. Queues controller cleanup asynchronously.
2. Keeps the offload engine object alive for normal driver rebind.
3. Disables VQs if the engine is enabled.
4. Disables the engine.
5. Stops VQs.
6. Flushes and unbinds CVQ if needed.
7. Stops and destroys the IO context.
8. Destroys VQs.
9. Stops the engine.
10. Leaves the engine object available unless this is full teardown.
For hot-unplug, the app intentionally does not destroy the engine on the reset edge alone. It waits for the later PCIe power-off / delayed-destroy path so that a reset during driver probe is not misclassified as final removal.
1.11. Teardown Flow
Full teardown performs stricter cleanup than reset:
1. Shut down the PCI config workqueue.
2. Stop stats and counter collection.
3. Destroy DevEmu Virtio-net counters.
4. Destroy queue debug state lists.
5. Destroy config MSI-X object if present.
6. Disable VQs.
7. Disable the offload engine.
8. Stop VQs.
9. Flush and unbind CVQ.
10. Stop and destroy IO context.
11. Destroy RX/TX/CVQ objects.
12. Stop the offload engine.
13. Destroy the DevEmu Virtio-net offload engine.
14. Stop and destroy TLP devices.
15. Destroy or close representors.
16. Stop and destroy the PCI type.
17. Tear down the DevEmu Virtio-net subsystem.
18. Remove the DOCA device from DevEmu Virtio-net.
19. Destroy progress engines and close the DOCA device.
1.12. Hotplug
In hotplug mode, endpoints are created and destroyed dynamically. The app accepts runtime commands such as plug and unplug. Hotplug work is performed on the worker path rather than directly on the TLP progress path.
For plug:
1. Create representor.
2. Create and start TLP device.
3. Create DevEmu Virtio-net controller/offload engine.
4. Update slot state.
5. Notify the host.
For unplug:
1. Mark unplug pending.
2. Signal hotplug state to the host.
3. Keep the device alive while the host resets/removes the driver.
4. Destroy the endpoint only after the host reaches the expected power-off/removal sequence.
1.13. Counters and Debug State
The DevEmu Virtio-net library exposes counters through `doca_devemu_vnet_counters`.
The reference app creates counters after the engine is enabled and periodically collects:
- RX virtqueue counters.
- TX virtqueue counters.
- Common VirtIO queue debug state.
The app also keeps a per-controller debug-state collection state . The queue debug-state data itself is stored in a `doca_devemu_virtio_queue_dbg_state` list. Each populated entry describes one VirtIO queue, including
the queue ID, enabled state, queue size, in-flight request count, hardware available/used indexes, and driver available/used indexes.
The app carefully stops collection before teardown to avoid destroying resources still in use by asynchronous population.
1.14. Live Update
The DevEmu Virtio-net offload engine supports export/import:
- `doca_devemu_vnet_offload_engine_export()`
- `doca_devemu_vnet_offload_engine_create_from_export()`
The reference VNet PCI application uses these APIs in '--lu-mode active' / '--lu-mode standby'. During handover, the active process exports the offload engine state and the standby process imports it, then starts the restored engine
and replays the required VirtIO queue state.The imported engine preserves runtime state such as MTU, MAC, queue count, and queue configuration.
The standby path skips normal TLP channel setup initially and restores channel/offload state through the live-update sequence.
1.15. Reference App Mental Model
A user reading `vnet_pci_dev` should view it as three cooperating subsystems:
`PCI/TLP emulator`: decodes host config/MMIO TLPs and returns completions quickly.
`VirtIO-net controller`: tracks features, device status, queue config, reset, hotplug, and CVQ commands.
`DevEmu Virtio-net offload integration`: creates offload engines and queues, enables packet datapath, exposes representor and counters.
The most important design rule in the reference app is that the main TLP progress path must stay responsive. Virtqueue startup, hotplug, reset cleanup, stats, and multiqueue scaling are deliberately moved away from the main PE to avoid host-visible PCIe timeouts.
Last updated: