Networking DOCA SDK Documentation

DOCA DevEmu Virtio-blk

This library is supported at alpha level; backward compatibility is not guaranteed.

Introduction

The DOCA TLP DevEmu Virtio-blk library is part of the DOCA DevEmu Virtio subsystem. It provides low-level APIs for creating and managing virtio-blk devices through the device-emulation capabilities of NVIDIA® BlueField® DPUs.

Architecture and Library Interaction

This library is part of a modular subsystem, where each library manages its own logic:

  • DOCA DevEmu PCI – Manages the PCIe transport layer, including configuration, discovery, and features like MSI-X and device hot plug/unplug.

  • DOCA DevEmu Virtio Common – Manages generic virtio logic, such as handling common virtio registers, the virtio offload engine, and virtqueue lifecycle. Note: for TLP-based virtio-blk devices, the virtio device reset flow is handled by the application.

  • DOCA DevEmu Virtio-blk – Manages the specific logic for the virtio block device, including block I/O request parsing, scatter-gather segment handling, and block device configuration.

Each library manages a separate layer of the emulation stack.

Core Functionality

The library converts virtio descriptors that contain block I/O requests from the device driver into doca_devemu_vblk_req requests and routes them to the application through registered event handlers. Each request contains the block operation, target logical block address (LBA), and a DOCA buffer that represents the host-memory data region. The library abstracts the device-specific acceleration details.

The library supports two data path providers:

  • DPA (Data Path Accelerator) — Offloads virtqueue processing to BlueField's DPA for hardware-accelerated, low-latency operation. Recommended for production workloads.

  • DPU (Arm cores) — Processes virtqueues on the DPU Arm cores for maximum flexibility and debugging.

User Responsibilities

An application that uses this library must implement a virtio-blk controller that performs the following tasks:

  • Manage the underlying DOCA DevEmu Virtio-blk offload engine and I/O contexts.

  • Receive the abstracted DOCA Virtio-blk requests via registered event handlers.

  • Process these requests according to the virtio-blk specification (read, write, get device ID, etc.).

  • Perform direct memory access (DMA) transfers between host and DPU memory as needed.

  • Complete each request back to the I/O context with the appropriate status code.

The backend block storage implementation is outside the scope of DOCA. The reference application (doca_vblk_pci_dev) demonstrates a RAM-backed block device for illustration.

Prerequisites

Before you use this library, read the following guides:

Environment

DOCA DevEmu Virtio-blk is supported on the BlueField target only. The BlueField must meet the following requirements:

  • DOCA version 3.4.0 or later

  • BlueField-3 firmware 32.49.0288 or later
    Note: See the DOCA Compatibility Policy.

  • Run applications that use this library with root privileges.

Perform the following steps:

  1. Configure BlueField to work in DPU mode as described in BlueField Modes of Operation.

  2. Enable TLP emulation by running the following on the DPU:

    [dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_ENABLE=1
    [dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set TLP_EMULATION_NUM_PF=32
    
  3. Perform a BlueField system cold reboot for the mlxconfig settings to take effect.

Hot-plug

Host Configuration

With a Linux environment on the host OS, additional kernel boot parameters are required to support the hot-plug feature:

  • For Intel machines: intel_iommu=on iommu=pt pci=realloc

  • For AMD machines: iommu=pt pci=realloc
    Note: On AMD machines, hotplug may not work.

Firmware Configuration

When PCIe switch emulation is enabled, BlueField can support hot-plugged virtio-blk functions. These PCIe functions are shared among all BlueField users and applications and may hold hot-plugged devices of type NVMe, virtio-blk, virtio-fs, or other (for example, virtio-net).

To enable PCIe switch emulation and configure hot-plugged ports, run:

[dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_ENABLE=1
[dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_TLP_PORT=1
[dpu] mlxconfig -y -d /dev/mst/mt41692_pciconf0 set PCI_SWITCH_EMULATION_NUM_PORT=2

PCI_SWITCH_EMULATION_NUM_PORT equals 1 plus the number of hot-plugged PCIe functions.

Architecture

The DOCA DevEmu Virtio-blk library provides the following main software abstractions:

  • The virtio-blk type – extends the PCIe TLP type; creates a PCI type specifically configured for virtio-blk device emulation

  • The virtio-blk offload engine – extends the virtio offload engine; manages virtqueue lifecycle and block-specific configuration (for example, seg_max)

  • The virtio-blk I/O context – extends the virtio I/O context; represents a progress context responsible for processing virtio descriptors carrying block I/O requests

  • The virtio-blk request – represents a single block I/O operation from the host driver

The following diagram illustrates the relationships between these abstractions:

image-2026-5-12_10-21-6.png

Virtio-blk Feature Bits

The following virtio feature bits are relevant to virtio-blk emulation:

Feature Bit

Description

VIRTIO_BLK_F_SIZE_MAX (bit 1)

Maximum segment size is reported

VIRTIO_BLK_F_SEG_MAX (bit 2)

Maximum segments per request is reported

VIRTIO_BLK_F_MQ (bit 12)

Multi-queue support

VIRTIO_F_INDIRECT_DESC (bit 28)

Indirect descriptor support

VIRTIO_F_VERSION_1 (bit 32)

VirtIO 1.0 compliance

VIRTIO_F_ACCESS_PLATFORM (bit 33)

Device access restricted to platform IOMMU

For TLP-based virtio-blk devices, feature bits are configured directly by the application in the emulated PCI configuration space. Indirect descriptor support must first be verified via doca_devemu_vblk_cap_is_indir_descs_supported() before enabling.

Virtio-blk Configuration Layout

According to the virtio specification, the virtio-blk device configuration structure layout includes the following key fields:

struct virtio_blk_config {
    le64 capacity;       /* device capacity in 512-byte sectors */
    le32 size_max;       /* maximum size of any single segment */
    le32 seg_max;        /* maximum number of segments in a request */
    /* ... additional fields ... */
    le16 num_queues;     /* number of request queues */
    /* ... */
};

For TLP-based virtio-blk devices, the application manages these fields directly in the emulated PCI configuration space. Configure seg_max and num_queues on the offload engine by calling doca_devemu_vblk_offload_engine_set_seg_max() and doca_devemu_virtio_offload_engine_set_num_queues(), respectively, so that the library can allocate the required resources.

Virtio-blk Type

The virtio-blk type creates a PCI TLP type specifically configured for virtio-blk device emulation, providing full control over the PCIe configuration space.

To create and configure a virtio-blk type:

  1. doca_devemu_vblk_is_pci_tlp_type_supported(): Check if the device supports virtio-blk TLP type management.

  2. doca_devemu_vblk_pci_tlp_type_create(): Create a stopped PCI TLP type for virtio-blk emulation.

  3. Configure the type using the following setters:

    • doca_devemu_pci_type_set_dev(): Associate a DOCA device with the type.

    • doca_devemu_pci_tlp_type_set_pci_cap_conf(): Register PCI capability configurations (for example, PCI Express, MSI-X).

    • doca_devemu_pci_type_set_num_msix(): Set the number of MSI-X vectors.

    • doca_devemu_pci_type_set_num_db(): Set the number of doorbells.

  4. doca_devemu_pci_type_start(): Start the PCI type to finalize configuration.

Virtio-blk Offload Engine

The virtio-blk offload engine (doca_devemu_vblk_offload_engine) extends the virtio offload engine with block-specific configuration. It is responsible for managing the state of all offloaded entities on a device, including configuration, creation, destruction, start, and stop of virtio-blk request virtqueues. Each offload engine is associated with a single DOCA DevEmu PCI endpoint.

Offload Engine Lifecycle

The offload engine follows this lifecycle. Configuration (for example, seg_max, indirect descriptors) happens between create() and start(). In handover and recovery scenarios, create_from_export() replaces both create() and the configuration phase — configuration values are inherited from the export descriptor:

image-2026-5-6_19-45-15.png
Standard Configuration

The standard flow for creating and configuring a virtio-blk offload engine is as follows:

  1. doca_devemu_vblk_offload_engine_create(): Create an offload engine bound to a PCI endpoint.

  2. doca_devemu_vblk_offload_engine_set_seg_max(): Set the maximum scatter-gather segments per request.

  3. Cast to virtio offload engine via doca_devemu_vblk_offload_engine_as_virtio_offload() for common configuration:

    • doca_devemu_virtio_offload_engine_set_indir_descs_enabled(): Enable/disable indirect descriptor processing.

    • doca_devemu_virtio_offload_engine_set_num_queues(): Set the number of virtqueues.

  4. doca_devemu_virtio_offload_engine_start(): Start the offload engine.

  5. doca_devemu_virtio_offload_engine_enable(): Enable the offload engine to begin processing requests.

Optional Configurations
  • doca_devemu_vblk_offload_engine_set_seg_max(): Set the seg_max value for the block device. If not set, the value defaults to the global seg_max configured via doca_devemu_vblk_set_seg_max().

  • doca_devemu_virtio_offload_engine_set_indir_descs_enabled(): Enable indirect descriptor support. Only valid if doca_devemu_vblk_cap_is_indir_descs_supported() returns true.

Export/Import Overlay

Function

When to Call

Effect

export()

After start(), before any VQ is started.

Produces an opaque export descriptor containing the runtime state of the offload engine. The caller owns the descriptor and must free it with free().

create_from_export()

Instead of create().

Creates a new idle offload engine seeded with the state from an export descriptor. The new engine must still be started and enabled.

Offload Engine States

A started offload engine is in one of three states:

image-2026-5-6_19-41-23.png

State

Description

DISABLED

Initial state after start(). No virtqueue operations are processed. Virtqueues cannot be individually enabled or disabled.

ENABLED

Processing operations from the host driver. Virtqueues can be individually enabled/disabled.

DISABLING

Transitioning to DISABLED. At least one virtqueue is still draining in-flight operations. Poll with get_state() until DISABLED.

Enable/Disable Semantics
  • enable() moves all started-and-disabled virtqueues to the enabled state. If the engine is already enabled, calling enable() again enables any newly started virtqueues that are currently disabled.

  • disable() may return DOCA_SUCCESS (synchronous) or DOCA_ERROR_IN_PROGRESS (asynchronous, at least one VQ is still draining). In the async case, poll with get_state() until the engine reaches DISABLED.

Handover Constraint

During a handover, for each source/destination pair of offload engines sharing the same PCI endpoint, at most one engine may be enabled at any time. Both engines may be disabled simultaneously (between the source's disable() and the destination's enable()), but having both enabled leads to undefined behavior.

State Export/Import (Live Upgrade and Recovery)

The DOCA DevEmu Virtio Block SDK provides a state export/import mechanism that enables two resilience scenarios for an offload engine managing a Virtio Block device:

Scenario

Description

Handover (Live Upgrade)

A running source process exports the offload engine state and transfers it to a destination process. The destination imports the state, and the two processes coordinate so that exactly one engine is enabled at any time. In-flight I/O during handover is not supported.

Takeover (Recovery)

A running process exports the offload engine state and stores the export descriptor in a location outside its own process that a recovering process can access. If the process crashes, a recovering process imports the descriptor to recreate the offload engine and replay in-flight I/O.

Both scenarios use the same three API calls:

doca_devemu_vblk_offload_engine_export()
doca_devemu_vblk_offload_engine_create_from_export()
doca_devemu_vblk_offload_engine_export_release()
API Reference

API

Description

Preconditions

Postconditions

doca_devemu_vblk_offload_engine_export()

Exports the runtime state of a started offload engine into an opaque descriptor blob.

  • The offload engine must be started.

  • No virtqueue associated with the engine may be started.

  • Must not be called concurrently.

  • export_desc points to an opaque blob owned by the caller.

  • The caller must free the descriptor buffer with free() when it is no longer needed.

doca_devemu_vblk_offload_engine_create_from_export()

Creates a new offload engine from an export descriptor. This is the import-side counterpart of export(). The created engine reflects the same runtime state as the exported engine once it is started and enabled.

  • export_desc must have been produced by doca_devemu_vblk_offload_engine_export().

  • The returned offload engine is idle (not started). It must be started and enabled to begin processing.

  • Configuration values (for example, seg_max, indirect descriptors) are inherited from the export descriptor.

doca_devemu_vblk_offload_engine_export_release()

Releases engine-side resources associated with the export. Must be called exactly once per export. Does not free the descriptor buffer — the caller is responsible for freeing it with free(). Double release is invalid and leads to undefined behavior.

  • export_desc must have been returned by export() and not yet released.

N/A

Handover (Live Upgrade) Flow

In a handover scenario, a source process transfers ownership of an offload engine to a destination process while maintaining host-visible state. The SDK does not prescribe any particular IPC mechanism or process architecture - the coordination protocol is application-defined.

  1. Source has a valid export descriptor. The export() call is typically done during regular bring-up of the application, not necessarily at the time of handover.

  2. Source transfers the export descriptor to the destination through an application-defined channel.

  3. Destination calls create_from_export() with the received descriptor and a PCI endpoint handle. This creates an idle offload engine seeded with the source's state.

  4. Destination calls start() on the new engine. Configuration is inherited from the descriptor. The engine enters the DISABLED state.

  5. Destination calls export() on its own engine to prepare for future recovery or live update. Virtqueues must not be started before this call.

  6. Destination creates, configures, and starts its virtqueues.

  7. Source calls disable() on its engine. If DOCA_ERROR_IN_PROGRESS is returned, poll get_state() until DISABLED.

  8. Once the source is fully disabled, destination calls enable() on its engine. The engine and its virtqueues must be enabled to start processing I/Os from the host driver.

  9. Source calls stop() and destroy() to release its engine.

  10. export_release() is called on the source's descriptor.

image-2026-5-6_20-18-16.png

Key invariant

At no point during the handover should both engines be enabled simultaneously.

Takeover (Recovery) Flow

In a takeover scenario, a process exports its offload engine state proactively so that a recovering process can restore the engine after a crash.

  1. Process A calls export() on its started offload engine and stores the export descriptor in a location outside its own process that a recovering process can access (for example, shared memory, file, or another process).

  2. Process A crashes.

  3. Process B (the recovering process) retrieves the stored export descriptor.

  4. Process B calls create_from_export() with the descriptor and a PCI endpoint handle. This creates an idle offload engine seeded with the crashed process's state.

  5. Process B calls start() on the engine.

  6. Process B creates, configures, and starts its virtqueues, then calls enable() on the engine. The engine and its virtqueues must be enabled to start processing I/Os. In-flight I/O from the recovery state is replayed.

image-2026-5-6_20-34-13.png

Process B must configure the virtio-blk global parameters (for example, doca_devemu_vblk_set_*()) with the same values that were used by Process A before takeover. These values are not captured in the export descriptor.

Request Virtqueue Management

Each virtio-blk virtqueue is created and managed through the offload engine:

API

Description

doca_devemu_vblk_req_vq_create()

Create a virtio-blk request virtqueue from a started offload engine. Must be called on the offload engine thread.

doca_devemu_vblk_req_vq_destroy()

Destroy a virtio-blk request virtqueue. Must not be started.

doca_devemu_vblk_req_vq_as_vq()

Cast to base virtio virtqueue for common operations (set_conf, start, stop, enable, disable).

Virtio-blk I/O Context

The virtio-blk I/O context (doca_devemu_vblk_io) extends the virtio I/O context. It is responsible for routing incoming block I/O requests from the host driver towards the application, and relaying completions back to the driver. Each I/O context is associated with a single offload engine and is progressed by a single DOCA Core progress engine. Usually, users configure a single virtio-blk I/O context per BlueField Arm core used by the application.

The I/O context can only route requests while in running state and when its associated offload engine is enabled.

Virtio-blk I/O Configurations

The flow for creating and configuring a virtio-blk I/O context is as follows:

  1. doca_devemu_vblk_io_create_from_offload_engine(): Create a new I/O context from a started offload engine. Must be called on the Arm core that will manage the I/O context.

  2. doca_devemu_vblk_io_event_vblk_req_register(): Register the event handler for incoming virtio-blk requests.

  3. Cast to DOCA context via doca_devemu_vblk_io_as_ctx() for progress engine connection:

    • doca_pe_connect_ctx(): Connect the I/O context to a progress engine.

    • doca_ctx_start(): Start the I/O context.

Mandatory Configurations
  • doca_devemu_vblk_io_event_vblk_req_register(): Register the event handler for incoming virtio-blk requests. This is mandatory.

Virtio-blk Request

The virtio-blk request object (doca_devemu_vblk_req) serves as an abstraction for handling block I/O requests arriving on virtio-blk request queues. These requests are generated by the device driver through virtio queues and routed to the user via the registered event handler on the associated I/O context.

Once the event handler is called, ownership of the virtio-blk request and the associated request user data transfers to the user. The ownership moves back to the associated I/O context when the request is completed by calling doca_devemu_vblk_req_complete().

The event handler provides the following parameters directly:

void handler(struct doca_devemu_vblk_req *req,
             uint32_t type,        /* VIRTIO_BLK_T_IN, VIRTIO_BLK_T_OUT, etc. */
             uint64_t sector,      /* starting sector (LBA) */
             void *req_user_data); /* per-request user data buffer */

The following APIs operate on a virtio-blk request:

API

Description

doca_devemu_vblk_req_get_data()

Get a DOCA buffer representing the data[] field of the virtio-blk request. This buffer represents the host memory for the data portion of the request according to the virtio specification.

doca_devemu_vblk_req_get_data_len()

Get the total data length (in bytes) of the data buffer linked list.

doca_devemu_vblk_req_get_data_list_len()

Get the number of scatter-gather elements in the data buffer linked list.

doca_devemu_vblk_req_get_vblk_io()

Get the associated DOCA Virtio Block I/O context.

doca_devemu_vblk_req_get_vq_user_data()

Get the user data associated with the virtqueue binding. Must be called on the I/O context thread.

doca_devemu_vblk_req_complete()

Complete the request with a status value and the number of bytes written into the device-writable portion of the buffer. The status follows the virtio specification: VIRTIO_BLK_S_OK (0), VIRTIO_BLK_S_IOERR (1), VIRTIO_BLK_S_UNSUPP (2).

Queue Counters

Each virtio-blk request virtqueue maintains a set of cumulative per-queue counters that cover generic virtio activity and block-specific data path events. The application reads them through a counters list object created on the offload engine.

Available Counters

The following counters are maintained per virtqueue:

Counter

Description

heads_fetched

Number of descriptor-chain heads fetched from the available ring

heads_completed

Number of descriptor-chain heads completed towards the driver

doorbells_handled

Number of doorbell events handled for the virtqueue

msix_raised

Number of MSI-X interrupts raised for the virtqueue

retry_count

Number of requests retried because the shared doca_buf inventory had no free buffers available

non_ok_status_count

Number of requests completed with a status other than VIRTIO_BLK_S_OK

hdr_dma_fail_count

Number of inbound DMA failures while fetching the request header or the indirect descriptor table

status_dma_fail_count

Number of outbound DMA failures while writing the status byte back to the host

The first four are common virtio counters, shared with other virtio device types. Read them with doca_devemu_virtio_queue_counters_get_<counter>() after casting the list element with doca_devemu_vblk_queue_counters_as_virtio_queue_counters().

The last four are virtio-blk specific. Read them with doca_devemu_vblk_queue_counters_get_<counter>() directly on the list element.

Every getter returns DOCA_ERROR_NOT_SUPPORTED if the virtqueue does not maintain that counter, and DOCA_ERROR_BAD_STATE if the element has not been populated.

Reading the Counters

Perform the following steps:

  1. doca_devemu_vblk_offload_engine_create_queue_counters_list(): Create a counters list on an enabled offload engine. The list keeps the offload engine enabled for as long as it exists.

  2. doca_devemu_vblk_queue_counters_populate_list(): Start an asynchronous population. Only one population may be in progress per offload engine.

  3. doca_devemu_vblk_queue_counters_is_populated(): Poll until the list reports that the population has finished.

  4. Read the values with the getters described above. Elements are addressed as (*counters_list)[queue_index].

  5. doca_devemu_vblk_queue_counters_destroy_list(): Destroy the counters list. Steps 2 to 4 may be repeated before the list is destroyed.

Each element holds the values captured by a single population, so the application reads a stable sample instead of accumulators that the data path keeps advancing.

In DPA provider mode, populating the four common counters requires a round trip to the device, because the DPA owns the data path that advances them. In DPU provider mode, and for any virtqueue that is not enabled, the values are sampled in place and the population completes without waiting.

Note: The counters are cumulative and there is no reset operation. They are stored in the virtqueue object, so a virtqueue that is destroyed and recreated, for example by a host driver reload, a PCI bind cycle, or an FLR, starts counting from zero. An application that compares samples across such an event must take a new baseline.

Counters List Management

API

Description

doca_devemu_vblk_offload_engine_create_queue_counters_list()

Create an unpopulated list of per-virtqueue counters for a started and enabled offload engine. Only a single counters list is allowed per offload engine.

doca_devemu_vblk_queue_counters_populate_list()

Issue an asynchronous population of the counters list. Returns DOCA_ERROR_IN_PROGRESS if a population is already in progress.

doca_devemu_vblk_queue_counters_is_populated()

Check whether the counters list has finished populating for all virtqueues.

doca_devemu_vblk_queue_counters_as_virtio_queue_counters()

Cast a virtio-blk queue counters element to the common virtio queue counters, for use with the common getters.

doca_devemu_vblk_queue_counters_get_qid()

Get the virtqueue index of a populated element.

doca_devemu_vblk_queue_counters_destroy_list()

Destroy the counters list. Returns DOCA_ERROR_IN_PROGRESS if a population is in progress.

Counter Capability Queries

API

Description

doca_devemu_vblk_cap_is_queue_counter_supported()

Check whether the device supports a given virtio-blk queue counter.

doca_devemu_vblk_cap_is_virtio_queue_counter_supported()

Check whether the device supports a given common virtio queue counter.

Global Configuration and Capability Queries

Before initializing the virtio-blk subsystem, the application must configure global parameters and may query device capabilities.

Global Configuration

The following APIs manage global configuration and the subsystem lifecycle. Call the configuration setters before doca_devemu_vblk_init(). For the setter APIs (set_vblk_req_user_data_size, set_max_queue_size, set_seg_max, set_datapath_on_dpa), the last value set before doca_devemu_vblk_init() takes effect.

API

Description

doca_devemu_vblk_add_dev()

Register a DOCA device for virtio-blk management. The device must support virtio-blk type management.

doca_devemu_vblk_rm_dev()

Unregister a previously added DOCA device.

doca_devemu_vblk_set_vblk_req_user_data_size()

Set the size of the per-request user data buffer allocated for each doca_devemu_vblk_req.

doca_devemu_vblk_set_max_queue_size()

Set the maximum virtqueue depth. Must conform to doca_devemu_vblk_cap_get_max_queue_size().

doca_devemu_vblk_set_seg_max()

Set the global maximum scatter-gather segments per request. Must conform to doca_devemu_vblk_cap_get_max_seg_max().

doca_devemu_vblk_set_datapath_on_dpa()

Set to true to run the data path on DPA, false for DPU Arm cores.

doca_devemu_vblk_init()

Initialize the virtio-blk subsystem. One-time call after all configuration is complete.

doca_devemu_vblk_teardown()

Tear down the virtio-blk subsystem and release all global resources.

Capability Queries

API

Description

doca_devemu_vblk_cap_get_max_queue_size()

Query the maximum supported virtqueue depth for the device.

doca_devemu_vblk_cap_get_max_seg_max()

Query the maximum supported seg_max value for the device.

doca_devemu_vblk_cap_is_indir_descs_supported()

Check whether the device supports indirect descriptor processing.

Discovery

The virtio-blk library uses the TLP emulation path, creating its own PCI type and representors dynamically.

To find and prepare a suitable device for virtio-blk emulation, perform the following steps:

  1. Open a DOCA device that supports virtio-blk TLP type management. This can be done by:

    • Iterating doca_devinfo_create_list() and checking doca_devemu_vblk_is_pci_tlp_type_supported() for each device, or

    • Opening a known device directly by IB device name (for example, mlx5_0) via helper utilities such as open_doca_device_with_ibdev_name().

  2. doca_devemu_vblk_pci_tlp_type_create(): Create the virtio-blk PCI TLP type.

  3. Configure and start the PCI type (doca_devemu_pci_type_set_dev(), doca_devemu_pci_type_start(), etc.).

  4. doca_devemu_pci_type_create_rep(): Create representors for each endpoint.

  5. doca_devemu_pci_tlp_dev_create(): Create PCI TLP devices from the representors.

At this point, the user can proceed with the initialization flow to create offload engines and I/O contexts.

Initialization

This section describes the initialization flow of a DOCA DevEmu Virtio-blk offload engine and one or more I/O contexts. During initialization, the user sets up and prepares the environment before starting to receive block I/O requests from the host.

Perform the following steps:

  1. Configure global virtio-blk parameters (add_dev, set_seg_max, set_datapath_on_dpa, etc.) and call doca_devemu_vblk_init().

  2. Choose N Arm cores to run the application I/O threads on.

  3. Create N DOCA Core progress engine (PE) objects.

  4. Create the virtio-blk PCI TLP type via doca_devemu_vblk_pci_tlp_type_create(), configure capabilities, and start it.

  5. Create a PCI TLP device for the endpoint.

  6. Configure global offload engine parameters.

  7. Create and configure the virtio-blk offload engine:

    •  If offload engine doesn't perform state import:

      • doca_devemu_vblk_offload_engine_create() with the PCI endpoint.

      • Configure seg_max, num_queues, and optionally indirect descriptors.

      • doca_devemu_virtio_offload_engine_start().

      • Optionally, call doca_vblk_pci_export() for future handover/export (see State Export/Import (Live Upgrade and Recovery)).

    • If state import:

      • doca_devemu_vblk_offload_engine_create_from_export() for future export, passing the export descriptor and its length along with the PCI endpoint.

      • If offload engine is a handover destination, call doca_devemu_vblk_offload_engine_export().

      • doca_devemu_virtio_offload_engine_start().

  8. Create N virtio-blk I/O contexts:

    • doca_devemu_vblk_io_create_from_offload_engine() on each Arm core.

    • Register the block request event handler.

    • Connect to the progress engine and start.

  9. Enable the offload engine via doca_devemu_virtio_offload_engine_enable().

At this point, the offload engine and I/O contexts are fully operational and ready to process block I/O requests from the host.

During the initialization flow, no block I/O requests are routed to the user until the offload engine is enabled.

Teardown

This section describes the teardown flow of a DOCA DevEmu Virtio-blk offload engine and its associated I/O contexts.

Perform the following steps:

  1. For each bound virtqueue, perform the per-VQ teardown sequence:

    1. doca_devemu_virtio_vq_disable(): Disable the virtqueue.

    2. If the call returns DOCA_ERROR_IN_PROGRESS (virtqueue is draining in-flight requests), optionally call doca_devemu_virtio_io_flush_vq() to flush pending events to the application.

    3. Handle and complete any outstanding virtio-blk requests via doca_devemu_vblk_req_complete().

    4. Loop: call doca_pe_progress() on the I/O context's progress engine until the virtqueue reaches DISABLED state (poll via doca_devemu_virtio_vq_get_state()).

    5. doca_devemu_virtio_io_unbind_vq(): Unbind the virtqueue from the I/O context.

  2. For each virtio-blk I/O context:

    • Stop the I/O context via doca_ctx_stop() and wait for idle state.

    • Destroy the I/O context via doca_devemu_vblk_io_destroy().

  3. Disable the offload engine via doca_devemu_virtio_offload_engine_disable() after all virtqueues are disabled.

  4. Destroy all virtqueues via doca_devemu_vblk_req_vq_destroy().

  5. Stop and destroy the offload engine:

    • doca_devemu_virtio_offload_engine_stop()

    • doca_devemu_vblk_offload_engine_destroy()

  6. Stop and destroy the PCI TLP type and devices.

  7. Tear down the virtio-blk subsystem via doca_devemu_vblk_teardown().

Execution Phase

This section describes execution on BlueField Arm cores using DOCA Core progress engines.

Control Path

The DOCA Virtio-blk offload engine manages the control path, including:

  • Virtqueue Lifecycle: Creation, configuration, start, enable, disable, stop, and destruction of request virtqueues, driven by host driver state transitions (for example, DRIVER_OK, device reset).

  • Device Reset: When the host driver writes device_status = 0, the virtio core library triggers a reset event. The application must tear down all active virtqueues, flush in-flight operations, and reinitialize the device.

  • Offload Engine State: The engine transitions through DISABLED → ENABLED → DISABLING → DISABLED, as described in Offload Engine States.

I/O Path

This section describes the flow for a single virtio-blk request sent by the device driver until its completion.

Before processing requests, register an event handler by calling doca_devemu_vblk_io_event_vblk_req_register().

The application must also understand the virtio-blk specification and be able to perform DMA operations between the host and local memory by using DOCA DMA or another suitable method.

The I/O path for a block request:

  1. Request arrival – The DOCA library invokes the registered event handler with the request object, type, sector, and user data.

  2. Request inspection – The application examines the request type:

    • VIRTIO_BLK_T_IN (0) – Read from block device.

    • VIRTIO_BLK_T_OUT (1) – Write to block device.

    • VIRTIO_BLK_T_GET_ID (8) – Return 20-byte device serial number.

    • Other types (for example, VIRTIO_BLK_T_FLUSH, VIRTIO_BLK_T_DISCARD) – The application decides how to handle these based on the feature bits it has exported to the host.

  3. Data access – The application calls doca_devemu_vblk_req_get_data() to obtain a DOCA buffer representing the host memory data region, and doca_devemu_vblk_req_get_data_len() for the total data length.

  4. DMA transfer – For read operations, the application populates a local buffer and uses DOCA DMA to transfer data to host memory. For VIRTIO_BLK_T_IN and VIRTIO_BLK_T_GET_ID requests, call doca_buf_reset_data_len() for each DOCA buffer segment before transferring data to host memory. For write operations, the application uses DOCA DMA to transfer data from host memory to a local buffer.

  5. Request completion – The application calls doca_devemu_vblk_req_complete(req, status, len) with the appropriate status code and the number of bytes written to the device-writable portion of the buffer.

Last updated: