DOCA Storage Initiator ComCh Application Guide

Introduction

The doca_storage_initiator_comch application is a client to the storage service and performs a benchmark of the performance in the process.

System Design

The doca_storage_initiator_comch application performs the following key functions:

• Initiates a series of IO requests to exercise the storage service

• Measures performance

  • Millions of IO operations per second

  • Effective IO data bandwidth

  • IO operation latency

    • Min

    • Max

    • Mean

To accomplish these tasks, the application establishes a connection to the service running on the NVIDIA® BlueField® platform using doca_comch_client.

Application architecture

The doca_storage_initiator_comch application is divided into two main functional areas:

• Control-time and shared resources

• Per-thread data path resources

The application execution follows two primary phases:

• Control phase

• Data path phase

Control Phase

This phase begins with establishing connections to the storage service. Once connected it starts issuing the comtrol commands to prepare the use-case and begin data transfers:

• Query storage

• Init storage

• Start storage

Issuing the start storage command initiates the data path phase. While the data threads begin execution, the main thread waits for the test execution to complete before finallt closing down the session by issuing the final control commands:

• Stop storage

• Shutdown

Data Path Phase

There are three data path routines that can be executed depending on mode:

1. Throughput test (read or write)

2. Read only data validity test

3. Write then read data validitiy test

Throughput test

In this mode tasks are submitted for the first them then a tight loop polling the progress engine is executed until the target operation count has been reached. As each task is submitted its assigned a memory location in the storage window to use; this location is then advanced ready for the next task to use, or if the end of storage is reached reset back to the start. In this way the memory is processed sequentially in a round robin fashion.

Read only data validity test

In this mode a copy of the expected storage memory is held by the initiator. tasks are submitted in th same was as with a throughput test but with the change being that once each segment of storage has been read the test is completed. As each task completes it compares the memory read from storage matches the data held in the relevant section of expected data file.

Write then read data validity test

This mode works start by writing a pattern accross the entire storage memory until each part of the storage memory has been "set". Following this the steps to perfom a read only data validity test are carried out but instead of using the content of a file loaded from disk to compare the read data to, the memory pattern that was generated to be written to the storage is compared against.

DOCA Libraries

This application leverages the following DOCA libraries:

• DOCA Comch

Compiling the Application

This application is compiled as part of the set of storage applications. For compilation instructions, refer to the DOCA Storage Applications page.

Running the Application

Application Execution

• Application usage instructions:

  Usage: doca_storage_initiator_comch [DOCA Flags] [Program Flags]
  
  DOCA Flags:
    -h, --help                        Print a help synopsis
    -v, --version                     Print program version information
    -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
    -j, --json <path>                 Parse all command flags from an input json file
  
  Program Flags:
    -d, --device                      Device identifier
    --cpu                             CPU core to which the process affinity can be set
    --storage-plain-content           File containing the plain data that is represented by the storage
    --execution-strategy              Define what to run. One of: read_throughput_test | write_throughput_test | read_write_data_validity_test | read_only_data_validity_test
    --run-limit-operation-count       Run N operations (per thread) then stop. Default: 1000000
    --task-count                      Number of concurrent tasks (per thread) to use. Default: 64
    --command-channel-name            Name of the channel used by the doca_comch_client. Default: "doca_storage_comch"
    --control-timeout                 Time (in seconds) to wait while performing control operations. Default: 5
    --batch-size                      Batch size: Default: 4
  
  
  

  This usage printout can be printed to the command line using the -h (or --help) options: 

  ./doca_storage_initiator_comch -h
  

  For additional information, refer to section "DOCA Storage Initiator ComCh Application Guide | id (3.0.0)DOCAStorageInitiatorComChApplicationGuide Command lineFlags".

• CLI example for running the application on the BlueField:

  ./doca_storage_initiator_comch -d 3b:00.0 --execution-strategy read_throughput_test --run-limit-operation-count 1000000 --cpu 0
  

  Both the DOCA Comch device PCIe address (3b:00.0) should match the addresses of the desired PCIe devices.

• The application also supports a JSON-based deployment mode in which all command-line arguments are provided through a JSON file:

  ./doca_storage_initiator_comch --json [json_file]
  

  For example:

  ./doca_storage_initiator_comch --json doca_storage_initiator_comch_params.json
  

  Before execution, ensure that the JSON file contains valid configuration parameters, particularly the correct PCIe device addresses required for deployment.

Command-line Flags

"log-level": 60

"sdk-log-level": 40

"device": "3b:00.0"

"execution-strategy": "read_throughput_test"

"cpu": 6

"storage-plain-content": "expected_data.txt"

"run-limit-operation-count": 1000000

"task-count": 64

"command-channel-name": "doca_storage_comch"

"control-timeout": 5

"batch-size": 4

Troubleshooting

Refer to the NVIDIA BlueField Platform Software Troubleshooting Guide for any issue encountered with the installation or execution of the DOCA applications.

Application Code Flow

Control Phase

1. initiator_comch_app app{parse_cli_args(argc, argv)};

   Parse CLI arguments, apply default values, and create the application instance.

2. C

   app.connect_to_storage_service();
   

   Connect to the storage service via doca_comch_client.

3. app.query_storage();

   Query the storage service so thats its capacity and block size can be known.

4. C

   app.init_storage();
   

   Prepare the storage service:

   1. Allocate memory resources.

   2. Send init storage control message.

   3. Create per thread resources including comch_consumer and comch_producer.

5. app.prepare_threads();

   Create thread objects and allocate per thread tasks

6. app.start_storage();

   Send start storage control message, wait for response then start data path threads

7. C

   app.run();
   

   Run the data path:

   1. Start threads.

   2. Wait for threads to complete.

   3. Collect and post-process statistics.

8. C

   app.join_threads();
   

   Join work threads.

9. C

   app.stop_storage();
   

   Send stop storage control message

10. C

    if (run_success) {
    		app.display_stats();
    } else {
    	exit_value = EXIT_FAILURE;
    	fprintf(stderr, "+================================================+\n");
    	fprintf(stderr, "| Test failed!!\n");
    	fprintf(stderr, "+================================================+\n");
    }
    

    Display execution result, or a failure banner if something wen't wrong during the data path

11. C

    app.shutdown();
    

    Send the shutdown control command to trigger storage service and storage targets to shutdown and release resources

Read/Write Throughput: data path

1. C

   while (hot_data.run_flag == false) {
   	std::this_thread::yield();
   	if (hot_data.error_flag)
   		return;
   }
   

   Wait for run flag to be set

2. C

   auto const initial_task_count = std::min(hot_data.transactions_size, hot_data.remaining_tx_ops);
   for (uint32_t ii = 0; ii != initial_task_count; ++ii)
   	hot_data.start_transaction(hot_data.transactions[ii], std::chrono::steady_clock::now());
   

   Submit initial tasks

3. C

   while (hot_data.run_flag) {
   	doca_pe_progress(hot_data.pe) ? ++(hot_data.pe_hit_count) : ++(hot_data.pe_miss_count);
   }
   

   Run until the test is complete

Read only data validity: data path

1. C

   while (hot_data.run_flag == false) {
   	std::this_thread::yield();
   	if (hot_data.error_flag)
   		return;
   }
   

   Wait for run flag to be set

2. C

   read_and_validate_storage_memory(hot_data, hot_data.storage_plain_content)
   

   Invoke the read and verify routine

   1. C

      hot_data.remaining_tx_ops = hot_data.remaining_rx_ops = io_region_size / hot_data.io_block_size;
      

      Set the expected op count to be 1 op per block

   2. 
      

      C

      auto const initial_task_count = std::min(hot_data.transactions_size, hot_data.remaining_tx_ops);
      for (uint32_t ii = 0; ii != initial_task_count; ++ii) {
      	char *io_request;
      	doca_buf_get_data(doca_comch_producer_task_send_get_buf(hot_data.transactions[ii].request), (void**)&io_request));
      	storage::io_message_view::set_type(storage::io_message_type::read, io_request);
      
      	hot_data.start_transaction(hot_data.transactions[ii], std::chrono::steady_clock::now());
      }
      

      Force all io_requests into read mode then start executing them

   3. C

      while (hot_data.remaining_rx_ops != 0) {
      	doca_pe_progress(hot_data.pe) ? ++(hot_data.pe_hit_count) : ++(hot_data.pe_miss_count);
      }
      

      Run until the test is complete

   4. C

      for (size_t offset = 0; offset != io_region_size; ++offset) {
      	if (hot_data.io_region_begin[offset] != expected_memory_content[offset]) {
      		DOCA_LOG_ERR("Data mismatch @ position %zu: %02x != %02x", offset, hot_data.io_region_begin[offset], expected_memory_content[offset]);
      		hot_data.error_flag = true;
      		break;
      	}
      }
      

      Validate memory content

Read write data validity: data path

1. C

   while (hot_data.run_flag == false) {
   	std::this_thread::yield();
   	if (hot_data.error_flag)
   		return;
   }
   

   Wait for run flag to be set

2. C

   size_t const io_region_size = hot_data.io_region_end - hot_data.io_region_begin;
   std::vector<uint8_t> write_data;
   write_data.resize(io_region_size);
   for (size_t ii = 0; ii != io_region_size; ++ii) {
   	write_data[ii] = static_cast<uint8_t>(ii);
   }
   

   Prepare new memory content

3. C

   write_storage_memory(hot_data, write_data.data());
   

   Invoke write memory routine

   1. C

      hot_data.remaining_tx_ops = hot_data.remaining_rx_ops = io_region_size / hot_data.io_block_size;
      

      Set the expected op count to be 1 op per block

   2. C

      hot_data.io_addr = hot_data.io_region_begin;
      std::copy(expected_memory_content, expected_memory_content + io_region_size, hot_data.io_region_begin);
      

      Place data to write to storage service into local memory

   3. auto const initial_task_count = std::min(hot_data.transactions_size, hot_data.remaining_tx_ops);
      for (uint32_t ii = 0; ii != initial_task_count; ++ii) {
      	char *io_request;
      	doca_buf_get_data(doca_comch_producer_task_send_get_buf(hot_data.transactions[ii].request), (void**)&io_request));
      	storage::io_message_view::set_type(storage::io_message_type::write, io_request);
      
      	hot_data.start_transaction(hot_data.transactions[ii], std::chrono::steady_clock::now());
      }

      Force all io_requests into write mode then start executing them

   4. C

      while (hot_data.remaining_rx_ops != 0) {
      	doca_pe_progress(hot_data.pe) ? ++(hot_data.pe_hit_count) : ++(hot_data.pe_miss_count);
      }
      

      Execute write operations

4. C

   read_and_validate_storage_memory(hot_data, write_data.data())
   
   
   

   Invoke the read and verify routine

   1. C

      hot_data.remaining_tx_ops = hot_data.remaining_rx_ops = io_region_size / hot_data.io_block_size;
      

      Set the expected op count to be 1 op per block

      C

      auto const initial_task_count = std::min(hot_data.transactions_size, hot_data.remaining_tx_ops);
      for (uint32_t ii = 0; ii != initial_task_count; ++ii) {
      	char *io_request;
      	doca_buf_get_data(doca_comch_producer_task_send_get_buf(hot_data.transactions[ii].request), (void**)&io_request));
      	storage::io_message_view::set_type(storage::io_message_type::read, io_request);
      
      	hot_data.start_transaction(hot_data.transactions[ii], std::chrono::steady_clock::now());
      }
      

      Force all io_requests into read mode then start executing them

   2. C

      while (hot_data.remaining_rx_ops != 0) {
      	doca_pe_progress(hot_data.pe) ? ++(hot_data.pe_hit_count) : ++(hot_data.pe_miss_count);
      }
      

      Run until the test is complete

   3. C

      for (size_t offset = 0; offset != io_region_size; ++offset) {
      	if (hot_data.io_region_begin[offset] != expected_memory_content[offset]) {
      		DOCA_LOG_ERR("Data mismatch @ position %zu: %02x != %02x", offset, hot_data.io_region_begin[offset], expected_memory_content[offset]);
      		hot_data.error_flag = true;
      		break;
      	}
      }
      

      Validate memory content

References

• /opt/mellanox/doca/applications/storage/

Last updated: April 18, 2025