DOCA Storage Target RDMA Application Guide

Introduction

The doca_storage_target_rdma application provides a simple, volatile memory-backed RDMA storage target. It is designed to interact with a DOCA storage service, offering direct read and write access to a dedicated memory region using RDMA.

System Design

The doca_storage_target_rdma application performs the following core functions:

• Exposes a memory-backed storage region for use by the storage service client.

• Handles RDMA-based I/O operations (reads and writes) using the DOCA RDMA library.

To perform these tasks, the application acts as a TCP server, waiting for an incoming connection from a storage service initiator. Once connected, it handles control and data path interactions as in this page.

Application architecture

The application is divided into two main functional areas:

• Control-Time and Shared Resources – Includes TCP server setup, memory registration, and RDMA connection handling.

• Per-Thread Data Path Resources – Includes thread-local RDMA resources and task management structures.

The application executes in two main phases:

• Control Phase

• Data Path Phase

Control Phase

This phase begins when a connection is established with a storage service over TCP. The application then processes a sequence of control commands:

1. Query Storage

   • Reports the size and layout of the exposed storage region.

2. Init Storage

   • Validates the requested number of worker threads.

   • Allocates and registers local memory for RDMA operations.

   • Imports remote memory handles provided by the initiator.

   • Creates internal worker objects for task execution.

3. Wait for RDMA Connection Requests

   • Waits for one RDMA connection per requested core/thread.

   • Ensures each RDMA session is fully established before proceeding.

4. Start Storage

   • Waits for all RDMA connections to be ready.

   • Submits initial tasks to prepare the data path phase.

   • Launches the data path threads.

Once the Start Storage command is received and all threads are active, the application transitions to the data path phase. The main thread remains active, waiting for final control commands:

1. Stop Storage

   • Terminates active data threads cleanly.

2. Shutdown

   • Performs cleanup and resource deallocation, shutting down the application.

Data Path Phase

Each data path thread performs I/O processing independently based on client requests. The typical per-thread flow is:

1. Receive I/O Request

   • Handle incoming requests from the initiator via RDMA.

2. Perform RDMA Operation

   • Depending on the request type, either:

     • Execute an RDMA read from the local memory region.

     • Execute an RDMA write to the local memory region.

3. Send I/O Response

   • Return a response back to the initiator, indicating success or failure of the operation.

DOCA Libraries

This application leverages the following DOCA libraries:

• DOCA RDMA

Compiling the Application

This application is compiled as part of the set of storage applications. For compilation instructions, refer to the DOCA Storage Applications page.

Running the Application

Application Execution

• Application usage instructions:

  Usage: doca_storage_target_rdma [DOCA Flags] [Program Flags]
  
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -v, --version                     Print program version information
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  
  Program Flags:
    -d, --device                      Device identifier
    --cpu                             CPU core to which the process affinity can be set
    --listen-port                     TCP listen port number
    --binary-content                  Path to binary .sbc file containing the initial content to be represented by this storage instance
    --block-count                     Number of available storage blocks. (Ignored when using content binary file) Default: 128
    --block-size                      Block size used by the storage. (Ignored when using content binary file) Default: 4096
  

  This usage printout can be printed to the command line using the -h (or --help) options: 

  ./doca_storage_target_rdma -h
  

  For additional information, refer to section "DOCA Storage Target RDMA Application Guide | id (3.1.0 core update)DOCAStorageTargetRDMAApplicationGuide Command lineFlags".

• CLI example for running the application:

  ./doca_storage_target_rdma -d 3b:00.0 --listen-port 12345 --block-size 4096 --block-count 64 --cpu 0
  

  The user DOCA device PCIe address (3b:00.0) should match the addresses of the desired PCIe device.

Command-line Flags

Troubleshooting

Refer to the NVIDIA BlueField Platform Software Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications.

Application Code Flow

Control Phase

The doca_storage_target_rdma application proceeds through the following control phase steps:

1. Parse CLI arguments and initialize the application:

   target_rdma_app app{parse_target_rdma_app_cli_args(argc, argv)};
   

   • Parses command-line arguments.

   • Applies default values.

   • Constructs the application instance.

2. Wait for client connection (TCP):

   app.wait_for_client_connection();
   

   • Blocks until a connection is established from the storage service application.

3. Process "Query Storage" request:

   app.wait_for_and_process_query_storage();
   

   • Waits for a query_storage control message.

   • Sends a response with the dimensions (e.g., size, block size) of the storage region.

4. Process "Init Storage" request:

   app.wait_for_and_process_init_storage();
   

   • Waits for an init_storage control message.

   • Performs the following:

   • Validates the requested core count does not exceed local resources.

   • Creates the local storage memory (doca_mmap).

   • Imports the remote memory provided by the initiator.

   • Initializes per-thread data path resources:

     • Worker objects

     • I/O message memory regions

     • doca_pe (progress engine) objects

     • doca_rdma objects

   • Sends an init_storage response.

5. Establish RDMA connections:

   app.wait_for_and_process_create_rdma_connections();
   

   • Waits for a series of create_rdma_connection control messages (one per thread).

   • For each:

     • Creates an exported connection descriptor using the local doca_rdma context.

     • Establishes the connection using the received remote connection details.

6. Start storage session:

   app.wait_for_and_process_start_storage();
   

   • Waits for a start_storage control message.

   • Ensures all RDMA connections are ready (fully negotiated).

   • Allocates task objects.

   • Submits receive tasks.

   • Launches worker threads.

7. Stop storage session:

   app.wait_for_and_process_stop_storage();
   

   • Waits for a stop_storage control message (signaling test completion).

   • Signals worker threads to stop.

   • Joins all worker threads.

8. Shutdown session:

   app.wait_for_and_process_shutdown();
   

   • Waits for a shutdown control message.

   • Collects and stores run-time statistics.

   • Cleans up and destroys all data path resources.

   • Sends a shutdown response.

9. Display results and cleanup:

   app.display_stats();
   

   • Outputs collected statistics (e.g., throughput, latency).

   • Destroys remaining control path resources.

Control Phase

1. target_rdma_app app{parse_target_rdma_app_cli_args(argc, argv)};

   Parse CLI arguments, apply default values, and create the application instance.

2. C

   app.wait_for_client_connection();
   

   Wait for a storage service app to connect via TCP.

3. C

   app.wait_for_and_process_query_storage();
   

   Wait for the storage service to send a query storage control message, then:

   • Send a query storage response with the dimmensions of this storage instance

4. C

   app.wait_for_and_process_init_storage();
   

   Wait for the storage service to send an init storage control message, then:

   • Verify that the requested core count does not exceed the available cores

   • Create local storage doca_mmap

   • Import remote memory doca_mmap

   • Create data path resources:

     • Worker objects

     • IO message memory regions

     • doca_pe objects

     • doca_rdma objects

   • Send an init storage response

5. C

   app.wait_for_and_process_create_rdma_connections();
   

   Wait for the storage service to send a number of create rdma connection control messages Processing each by:

   • Create a connection by creating an exported connection details using doca_rdma context specified in the command

   • Starting the connection using the provided remote connection details

6. C

   app.wait_for_and_process_start_storage();
   

   Wait for the storage service to send a start storage control message, then:

   • Wait until all doca_rdma contexts are ready to execute tasks (both sides have started their respective connections)

   • Allocate task objects

   • Submit receive tasks

   • Create and start worker threads

7. C

   app.wait_for_and_process_stop_storage();
   

   Wait for the storage service to send a stop storage control message (test complete), then:

   • Signal worker threads to stop

   • Join worker threads

8. C

   app.wait_for_and_process_shutdown();
   

   Wait for the storage service to send a shutdown control message, then:

   • Collect and store run statistics

   • Destroy data path objects

   • Send a shutdown response

9. C

   app.display_stats();
   

   Display collected statistics and destroy all control path objects.

References

• /opt/mellanox/doca/applications/storage/

Last updated: July 29, 2025