DOCA Perftest

This guide describes DOCA Perftest, an RDMA benchmarking tool designed for compute clusters that enables fine-tuned evaluation of bandwidth, message rate, and latency across various RDMA operations and complex multi-node scenarios.

Introduction

NVIDIA® DOCA Perftest is an RDMA benchmarking utility designed to evaluate performance across a wide range of compute and networking environments (from simple client-server tests to complex, distributed cluster scenarios).

It provides fine-grained benchmarking of bandwidth, message rate, and latency, while supporting diverse RDMA operations and configurations.

Key features:

• Comprehensive RDMA Benchmarks – Supports bandwidth, message rate, and latency testing.

• Unified RDMA Testing Tool – A single executable for all RDMA verbs, with rich configuration options and CUDA/GPUDirect RDMA integration.

• Cluster-Wide Benchmarking – Run distributed tests across multiple nodes, initiated from a single host, with aggregated performance results.

• Flexible Scenario Definition – Define complex multi-node, multi-test configurations via a JSON input file.

• Command-Line Simplicity – Quickly run local or point-to-point benchmarks directly from the CLI.

• Synchronized Execution – Ensures all benchmarks begin and end simultaneously for consistent results.

The DOCA Perftest utility simplifies evaluation and comparison of RDMA performance across applications and environments.

Comparison with Legacy Perftest

Unlike legacy RDMA benchmarking tools (e.g., ib_write_bw, ib_send_lat), DOCA Perftest is a native implementation designed for modern data centers. As opposed to a wrapper, it is a standalone product that replaces both the legacy tools and the custom orchestration scripts often required to run them at scale.

Architectural differences:

Benefits of migration to DOCA Perftest:

• Standard RDMA benchmarks require complex external scripts (Ansible, Bash, Python) to manage remote process launching, NUMA pinning, GPU selection and result parsing. DOCA Perftest handles these natively via the CLI or JSON scenario files.

• In large-scale clusters, measuring fabric congestion or incast/outcast scenarios requires all nodes to hit the network simultaneously. DOCA Perftest utilizes a centralized sync engine to ensure all processes begin and end traffic in a coordinated window, providing accuracy that is impossible to achieve with asynchronous legacy wrappers.

• While legacy tools require running multiple instances to saturate high-speed links (e.g., 200G/400G+), DOCA Perftest scales linearly across cores within a single execution using the -N or -C flags.

• Rather than collecting individual output files from dozens of servers, DOCA Perftest provides a unified report. This includes the full scenario definition, all raw results, and calculated aggregations per-device, per-node, and per-test.

Setup and Dependencies

DOCA Perftest is included in the DOCA Networking installation profile. It requires only a standard RDMA stack to run, with optional DOCA SDK components enabling advanced capabilities.

RDMA Core (libibverbs)

The only mandatory dependency is libibverbs (part of the rdma-core package). DOCA Perftest supports both the version bundled with the DOCA deployment and the upstream open-source release from rdma-core.

MPI (Multi-Node Scenarios)

Multi-node scenarios (JSON mode and --traffic_pattern CLI mode) require OpenMPI for orchestration.

DOCA Perftest relies on the OpenMPI package bundled with DOCA.

MPI is not required for simple point-to-point CLI benchmarks.

Optional DOCA SDK Components

Depending on the DOCA installation, additional capabilities are available:

• DOCA Verbs (-r dv) – The DOCA RDMA Verbs backend provides a high-performance alternative to standard libibverbs. Loaded on demand; if the package is not installed, DOCA Perftest operates normally with the default IBV driver.

• DOCA GPUNetIO (--engine gpunetio) – Offloads the RDMA data-path to GPU CUDA kernels using DOCA GPUNetIO. Requires the doca-gpunetio package, CUDA Toolkit, and a Volta+ GPU. For details, see GPUNetIO Engine (GPU-Initiated RDMA).

If an optional component is not installed and a user explicitly requests it (e.g., -r dv without DOCA Verbs), doca_perftest exits with a descriptive error message.

Point-To-Point Benchmarks

For simple benchmarks, run doca_perftest directly from the command line.

When invoked on the client, the utility automatically launches the corresponding server process (requires passwordless SSH) and selects optimal CPU cores on both systems based on NUMA affinity.

Example command:

# Run on client
doca_perftest -d mlx5_0 -n <server-host-name>

This is equivalent to running:

# On server
doca_perftest -d mlx5_0 -N 1 -c RC -v write -m bw -s 65536 -D 10

# On client
doca_perftest -d mlx5_0 -N 1 -c RC -v write -m bw -s 65536 -D 10 -n <server-host-name>

Parameter breakdown:

Scenario-Based Benchmarks

When a benchmark involves multiple devices, multiple nodes, or multiple test configurations, DOCA Perftest uses MPI to orchestrate execution across all participants. Tests are launched, synchronized, and results aggregated automatically on the initiating host.

Examples of scenarios that require orchestration:

• Multi-node – Traffic between two or more hosts across the network.

• Multi-device – Traffic across multiple NICs on the same host or across hosts (e.g., railed configurations with mlx5_0 and mlx5_1).

• Multi-test – Several benchmark configurations running simultaneously within a single execution — for example, mixed message sizes ("elephant and mice"), latency under background load, or short-haul vs long-haul comparisons.

Scenario-based benchmarks can be configured in two ways:

• CLI mode (--traffic_pattern) – Define a single test directly from the command line. No configuration file required.

• JSON mode (-f) – Define one or more tests in a JSON configuration. Required for multi-test scenarios.

A test is a single benchmark configuration: one verb, one message size, one traffic pattern, and one set of participating nodes. CLI mode runs exactly one test; JSON mode can define multiple tests that execute in parallel, synchronized to start and stop together.

CLI Scenario Mode

Use the --traffic_pattern flag to launch an orchestrated scenario directly from the command line:

# Basic: all-to-all across 12 nodes on mlx5_0
doca_perftest -d mlx5_0 -n my_node_0[1-12] --traffic_pattern all_to_all

# Advanced: railed all-to-all on two NICs, 10 QPs, 1KB messages, 60-second run
doca_perftest -d mlx5_[0,1] -n my_node_0[1-12] --traffic_pattern a2a --traffic_partition railed -q 10 -s 1024 -D 60

This mode of work builds a scenario configuration from CLI flags and feeds it into the standard MPI pipeline. All results, progress tracking, and aggregation work identically to JSON mode.

The advanced example runs an all-to-all pattern across 12 nodes using two NICs per node (mlx5_0 and mlx5_1). The railed partition ensures each NIC only communicates with the same-named NIC on peer nodes (e.g., mlx5_0 ↔ mlx5_0). The test uses 10 queue pairs, 1 KB messages, and runs for 60 seconds.

Key flags:

All standard test parameters (-v, -m, -s, -D, -q, etc.) are supported alongside --traffic_pattern.

JSON Scenario Mode

For multi-test scenarios or advanced configurations, provide a JSON scenario definition with -f / --input_file:

• From file – Load from a JSON file. Scenario files can be version-controlled, shared across teams, and reused for regression testing.

• Inline string – Pass a raw JSON string directly (wrap in single quotes). Useful for automation pipelines where the configuration is generated programmatically, avoiding extra file I/O.

# From file
doca_perftest -f path_to_scenario_file.json

# From inline JSON string
doca_perftest -f '{"testNodes": [...], "trafficPattern": "ALL_TO_ALL"}'

JSON mode capabilities:

• Can be initiated from any node in the cluster (even non-participating ones).

• All benchmarks run simultaneously, with millisecond-level synchronization for benchmark start and stop across all nodes.

• Supports all traffic patterns and partitions.

• Fully compatible with all CLI parameters (JSON parameters inherit the same defaults).

Common multi-test scenarios:

• Mixed message sizes – Combine a large-message bandwidth test with a small-message message-rate test to evaluate how the fabric handles concurrent bulk and signaling traffic ("elephant and mice").

• Latency under load – Run bandwidth traffic as background noise while measuring latency on a separate set of connections, revealing how congestion affects tail latency.

• Short-haul vs long-haul – Define one test for intra-rack traffic (low-hop, optimized for latency) and another for inter-rack or cross-fabric traffic (different RDMA parameters tuned for distance).

• Message size sweep – Run tests at multiple message sizes (e.g., 64B, 4KB, 64KB, 1MB) in a single scenario to profile throughput across the full range.

Results and Output

By default, DOCA Perftest prints aggregated results to the terminal at the end of execution. This summary includes the total bandwidth, message rate, or latency statistics across all connections.

For detailed, machine-readable results, use the outputFile field (JSON mode) or the -j flag (CLI mode) to produce a structured JSON report. The JSON output provides full visibility into every level of the test hierarchy:

• Per-connection results – Raw results for each individual connection, broken down by process. This level of detail is critical for identifying anomalies: if a subset of connections underperforms due to congestion, routing issues, or hardware faults, the per-connection data reveals it (even when the aggregated summary appears healthy).

• Per-device and per-test aggregations – Results rolled up by NIC and by benchmark, making it easy to compare device-level performance across a cluster.

• Per bidirectional connection aggregation (BIDIR tests only) – For each bidirectional connection pair, the two per-direction results are combined into a single entry under aggregatedResults.bidirConnections (BW and message rate are summed across the two directions). The terminal report also includes a "BIDIR Connections Summary" section with the overall average, minimum, and maximum BW across all bidirectional pairs (the min/max rows identify which connection contributed each value). The per-direction results are also available in the top-level connections array.

• Full scenario configuration – The exact parameters used for execution, enabling reproducibility.

• Threshold pass/fail status (if configured) and QP histogram data (if enabled).

# JSON mode: add to scenario config
"outputFile": "results.json"

# CLI mode: save results to file
doca_perftest -d mlx5_0 -n server-name -j results.json

# CLI mode: default results path
doca_perftest -d mlx5_0 -n server-name -j

The JSON output schema is versioned. Schema version and DOCA Perftest version are included in the output under the versionInfo block.

For programmatic consumption of results via stdout, see Structured JSON Output to stdout.

Benchmark Results

Bandwidth

Bandwidth tests measure the aggregate data transfer rate and message-handling efficiency across all participating processes.

Metrics collected:

• Message Rate (million operations per second) – The total number of Completion Queue Entries (CQEs) processed per second across all test processes. This is an operation/completion rate; one CQE does not necessarily correspond to one packet on the wire.

• Bandwidth (Gigabits per second) – The application-level data transfer rate, calculated from completed benchmark operations tracked through CQEs and the configured message size (-s or --msg_size). This metric evaluates the system’s ability to sustain high-throughput communication.

Measurement notes:

• Concurrency Handling – Results reflect the sum of completed-operation counts derived from CQEs across all concurrent test processes, as specified by the -C command-line argument or --cores field in the input JSON file.

• Test duration – The duration is averaged across all test processes, ensuring consistency in measuring sustained performance over time.

In multi-node scenarios, bandwidth results are aggregated at multiple levels: per-process, per-device, per-node, and per-test. The JSON output includes all aggregation levels for post-processing.

Interpretation tips:

Latency

Latency tests measure RDMA operation delay using a ping-pong protocol. By default, both sides participate in the measurement loop: the requestor sends data, the responder receives it and sends data back, and the requestor reports half of the measured round-trip time (RTT/2). This is not a separately clock-synchronized endpoint-to-endpoint one-way measurement.

Latency Modes

Two measurement modes are available, selected via the --lat_mode CLI flag or the "latencyMode" JSON field:

# Full round-trip latency (default)
doca_perftest -d mlx5_0 -n server-name -m lat -v write

# One-way mode (local send-completion timing)
doca_perftest -d mlx5_0 -n server-name -m lat -v send --lat_mode one_way

JSON configuration:

"metric": "lat",
"latencyMode": "full_round_trip"

Verb Compatibility

All RDMA verbs support full round-trip latency by default. The one-way mode has additional restrictions:

Metrics Collected

• Minimum latency – Fastest observed transaction

• Maximum latency – Longest observed transaction

• Mean latency – Average across all iterations

• Median latency – Midpoint value (less influenced by outliers)

• Standard deviation – Variability indicator

• 99% tail latency – 99% of messages completed within this time

• 99.9% tail latency – Outlier detection for extreme cases

Measurement Notes

• Full round-trip mode uses a tight ping-pong loop where both sides alternate sending and receiving.

• Timing is collected on the requestor side.

• In one_way mode, only the requestor posts and polls its send completion queue. The responder does not send data back; the result is requestor-side operation-completion time, not endpoint-to-endpoint one-way network latency.

• Aggregated across processes for final reporting.

Interpretation tips:

Common Arguments and Use-Cases

This section highlights some of the most commonly used parameters and use-cases.

Traffic Directions

DOCA Perftest supports three traffic directions that determine how data moves between nodes and how queue pairs are allocated.

Unidirectional Traffic (Default)

• In unidirectional mode, traffic flows in one direction only.

• The client (requestor) initiates operations, and the server (responder) receives them.

• This is the default mode and provides clear, predictable performance metrics.

Bidirectional Traffic

In bidirectional mode, traffic flows in both directions simultaneously. Each side acts as both requestor and responder, creating full-duplex communication.

Bidirectional tests use two traffic runners (requestor + responder) sharing resources. It may show different aggregate bandwidth than 2× unidirectional. Bidirectional mode is supported for Write, Send and Atomic verbs.

Run bidirectional traffic from the command line:

# Enable bidirectional traffic
doca_perftest -d mlx5_0 -n <server-name> -b

Dual Unidirectional Traffic

Dual unidirectional mode runs two simultaneous unidirectional flows in opposite directions.
Unlike BIDIR, each direction uses its own connection and dedicated QP, and each endpoint retains a
single requestor or responder role per connection.

Run dual unidirectional traffic in CLI scenario mode:

doca_perftest -d mlx5_0 -n 'node[01-02]' --traffic_pattern one_to_one --dual_unidir

The --dual_unidir flag requires --traffic_pattern and is mutually exclusive with --bidirectional.
In JSON mode, set "trafficDirection" to "DUAL_UNIDIR". The other supported JSON values are
"UNIDIR" and "BIDIR".

Traffic Patterns

Traffic patterns provide built-in shortcuts for complex multi-node communication scenarios.

While these configurations were always possible through detailed JSON definitions, traffic patterns dramatically simplify setup for common topologies.

Example JSONs using traffic patterns are available under /usr/share/doc/doca-perftest/examples.

Available patterns:

• ONE_TO_ONE

• ONE_TO_MANY

• MANY_TO_ONE

• ALL_TO_ALL

• BISECTION

• RING

They collapse complex multi-node wiring into a few lines of JSON. Instead of manually listing dozens of connections, you specify a regex-like host list and a pattern (e.g., ALL_TO_ALL) and DOCA Perftest generates and synchronizes all connections for you.

One-to-One (O2O)

Simple point-to-point between two nodes; useful for baseline performance testing.

"testNodes": [ {"hostName": "node01", "deviceName": "mlx5_0"},
               {"hostName": "node02", "deviceName": "mlx5_0"} ],
"trafficPattern": "ONE_TO_ONE"

One-to-Many (O2M)

Single sender to multiple receivers; the first node sends to all others.

"testNodes": [ {"hostName": "sender", "deviceName": "mlx5_0"},
               {"hostName": "receiver[1-10]", "deviceName": "mlx5_0"} ],
"trafficPattern": "ONE_TO_MANY"

This creates 10 connections: sender→receiver1, sender→receiver2, ..., sender→receiver10.

Many-to-One (M2O)

Multiple senders to one receiver; all nodes send to the first node.

"testNodes": [  {"hostName": "aggregator", "deviceName": "mlx5_0"},
                {"hostName": "client[01-20]", "deviceName": "mlx5_0"}  ],
"trafficPattern": "MANY_TO_ONE"

This creates 20 connections: client1→aggregator, client2→aggregator, ..., client20→aggregator.

All-to-All (A2A)

Full-mesh connectivity; every node connects to every other node.

"testNodes": [  {"hostName": "compute[01-16]", "deviceName": "mlx5_0"} ],
"trafficPattern": "ALL_TO_ALL",
"trafficDirection": "UNIDIR"

This creates 240 connections (16×15) for UNIDIR or DUAL_UNIDIR, or 120 bidirectional pairs for
BIDIR.

Variable All-to-All (per-pair message sizes)

Standard ALL_TO_ALL uses a single message size for every directional pair. For workloads where each (source, dest) connection moves a different volume of data (e.g. UCC ALLTOALLV, MoE expert routing), the per-pair size can be supplied via the ptiniFile parameter, pointing at a UCC ALLTOALLV pattern file (.ptini) as produced by UCC's own benchmark tool, ucc_perftest. The values in such a file are element counts, and ptiniDatatype names the element type they are scaled by (default float32).

JSON:

"trafficPattern": "ALL_TO_ALL",
"trafficDirection": "UNIDIR",
"ptiniFile": "/path/to/sizes.ptini",
"ptiniDatatype": "float32"

CLI scenario mode:

doca_perftest --traffic_pattern all_to_all -n compute[01-05] -d mlx5_0 \
              --ptini_file /path/to/sizes.ptini --ptini_datatype float32 -D 10

~/... paths are expanded against $HOME.

File format

The file is a UCC (openucx/ucc) ALLTOALLV pattern file (the same format ucc_perftest consumes):

[format]
version = 1.0

[collective]
type = ALLTOALLV
team_size = 5
flags = COUNT_32BIT
counts = {
    0,        33554432, 262144,   2097152,  524288
    33554432, 0,        1048576,  0,        524288
    262144,   1048576,  0,        16777216, 524288
    2097152,  0,        16777216, 0,        524288
    2097152,  0,        16777216, 1048576,  0
}

Inside counts, each row is a source rank and each column is a destination rank. Values are element counts, not bytes: exactly as in UCC, the wire size of a pair is its count multiplied by the size of the datatype. The file itself does not carry the datatype; ucc_perftest takes it on its command line via -d, and its file reader never looks for a datatype key. ptiniDatatype / --ptini_datatype is the DOCA Perftest equivalent of that flag, and any datatype = ... line inside a .ptini file is ignored.

ptiniDatatype accepts the UCC predefined datatype names (int8, uint8, int16, uint16, float16, bfloat16, int32, uint32, float32, float32_complex, int64, uint64, float64, float64_complex, int128, uint128, float128, float128_complex) and defaults to float32 (4 bytes). That default is ucc_perftest's own default, so replaying a capture without naming a datatype produces the same message sizes UCC would have produced. With it, the matrix above spans 1 MiB to 128 MiB. Specify int8 if you want the counts to be plain byte counts.

type, team_size, and counts are interpreted; datatype, flags, and the [format] section are ignored. type must be ALLTOALLV and team_size must equal the rank count when either is present. Line breaks inside counts are cosmetic (UCC flattens the block into a single list) and ; or # begin a comment.

For N ranks the block must hold exactly N × N values. Diagonal entries are ignored. Zero off-diagonal entries skip that directional pair (no traffic, no QP). A file holding more than one [collective] pattern is rejected: UCC cycles through every pattern within a run, while a DOCA Perftest test benchmarks exactly one, so such a file must be split.

The expanded testNodes list (after hostlist + device-list + cudaDeviceId expansion) defines the rank ordering: rank i is the i-th entry of the expanded list.

Constraints

• Only ALL_TO_ALL / A2A. Other patterns reject ptiniFile.

• UNIDIR only, in every mode (including iterationSync). BIDIR + ptiniFile is rejected because a BIDIR
  connection carries one msgSize for both directions and would collapse asymmetric pair sizes. With UNIDIR,
  each directed pair is independent. Iteration-sync connections report their own BW; the combined synchronized
  result omits aggregate bwResults and displays N/A.

• msgSize, operations, and ptiniFile are mutually exclusive.

• ptiniDatatype requires ptiniFile; specifying it on its own is rejected.

• bwThreshold is not supported with variable-A2A iterationSync because bandwidth is unavailable.

• verb is independent and can be combined with ptiniFile.

Visualizing the matrix

When you set ptiniFile, you see the parsed matrix before any traffic starts, so you can confirm the run will send the sizes you meant it to. Rows are the sending endpoints and columns the receiving ones, named the same way the connection matrix report names them.

Total sent and Total recv close the table with each endpoint's row and column sums, so you can tell at a glance which NIC carries the most bytes. The diagonal shows -, a pair that carries no traffic shows 0, and every other cell shows its byte count. The line above the table names the datatype the .ptini counts were scaled by, so you can check the scaling that produced those bytes.

Test: VariableA2A, endpoints: 5
Hosts: gb-ida-lr1-r8-*
.ptini counts scaled by datatype float32 (4 bytes per element)
Message Size Matrix [bytes]:
             |    01     |    02     |    03     |    04     |    05     |   Total
             |  mlx5_0   |  mlx5_0   |  mlx5_0   |  mlx5_0   |  mlx5_0   |   sent
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
01    mlx5_0 |     -     | 134217728 |  1048576  |  8388608  |  2097152  | 145752064
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
02    mlx5_0 | 134217728 |     -     |  4194304  |     0     |  2097152  | 140509184
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
03    mlx5_0 |  1048576  |  4194304  |     -     | 67108864  |  2097152  | 74448896
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
04    mlx5_0 |  8388608  |     0     | 67108864  |     -     |  2097152  | 77594624
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
05    mlx5_0 |  8388608  |     0     | 67108864  |  4194304  |     -     | 79691776
----- ------ | --------- | --------- | --------- | --------- | --------- | ---------
Total recv   | 152043520 | 138412032 | 139460608 | 79691776  |  8388608

This example omits (2→4), (4→2), and (5→2), leaving 17 of 20 directional
connections; the parser logs skipped 3/20 zero-size pairs.

The output is stdout-only and is never written to the results JSON. The per-pair msgSize values survive into connections[*].testConfig.resourceConfig.operations[0].msgSize in the results JSON, so the same totals can be reconstructed programmatically by grouping the connections array.

Runnable configurations are provided in input/examples/all2all_v_example.json and input/examples/all2all_v_iter_sync_example.json.

The JSON output schema version (versionInfo.jsonOutputSchema) is 1.4.0. Schema 1.3.0 added ptiniFile and
ptiniDatatype to the serialized general config; ptiniDatatype is empty for runs that do not use a .ptini
file.

Bisection (B)

Divides nodes into two equal halves; the first half connects to the second half. Requires an even number of nodes.

"testNodes": [  {"hostName": "rack1-[01-10]", "deviceName": "mlx5_0"},
                {"hostName": "rack2-[01-10]", "deviceName": "mlx5_0"} ],
"trafficPattern": "BISECTION"

This creates 10 connections: rack1-01↔rack2-01, rack1-02↔rack2-02, ..., rack1-10↔rack2-10.

Ring (R)

Each node connects to the next in a circular ring: node i sends to node (i+1) % N. Requires at least 2 nodes.

"testNodes": [  {"hostName": "compute[01-08]", "deviceName": "mlx5_0"} ],
"trafficPattern": "RING"

This creates 8 connections: compute01→compute02, compute02→compute03, ..., compute08→compute01. All connections run simultaneously, not sequentially.

Traffic Partitions

When nodes have multiple NICs, a traffic partition specifies which connections to establish based on the relationship between devices and hosts. Combined with a traffic pattern, it provides a concise way to describe common multi-device topologies in a single test definition.

Set via the "trafficPartition" JSON field or the --traffic_partition CLI flag.

Example — railed all-to-all across a cluster with two NICs per node:

"testNodes": [  {"hostName": "node[01-16]", "deviceName": "mlx5_[0,1]"} ],
"trafficPattern": "ALL_TO_ALL",
"trafficPartition": "RAILED"

Without the partition, achieving the same railed topology would require a separate test for each device with otherwise identical parameters.

Traffic partitions apply to any traffic pattern. The partition determines which connections are kept regardless of how they were generated. Some pattern-partition combinations may result in zero connections (for example, LOCAL with a single device per node, or REMOTE on a one-to-one test within the same host).

Per-iteration-sync Flow (Lock-step Benchmarking)

Designed to mimic AI workloads, this flow ensures data transfer occurs in distinct, synchronized steps. By forcing every process to wait for all peers to complete an iteration before proceeding, it enables granular data validation and allows for QP parameter modification between steps.

Configuration constraints:

• Requires the ALL_TO_ALL pattern. Supports both UNIDIR and BIDIR traffic: in BIDIR both endpoints of a connection exchange data each iteration; in UNIDIR each directed pair (i→j) is its own connection and only the requestor sends data while the responder receives. UNIDIR is required for variable-A2A (ptiniFile) so each pair can use its own message size. The lock-step sync handshake (zero-length RDMA Write with Immediate) is symmetric and runs in both directions regardless.

• Must be defined by specific iterations (time-based duration is not supported).

• Supported in both JSON mode ("iterationSync": true) and CLI mode (--traffic_pattern a2a --iteration_sync).

Logic and Implementation

The flow utilizes an ALL_TO_ALL pattern (UNIDIR or BIDIR). Each iteration consists of four distinct phases:

1. Data phase:

   1. Every sender sends a data message to its peers (in BIDIR both endpoints send;
      in UNIDIR only the requestor of each directed pair sends, while the responder receives).

   2. The total msgSize is split across available QPs. Each QP writes to a specific offset to utilize the full buffer.

2. Sync phase:

   • Once data transfer completes, each process sends a Sync Message to all peers.

   • The Sync Message is a zero-length RDMA Write with Immediate Data.

3. Barrier phase:

   • A process completes the iteration only after it has received confirmation for its own Sync Send and received Sync Messages from all peers.

4. Post-iteration (management) phase:

   • Occurs after synchronization but before the next iteration begins.

   • Performs non-timed management tasks, such as modifying QPs, checking data validation results, or updating pointers.

Configuration

Configure iteration-sync in JSON (per-test fields below) or via CLI (--traffic_pattern a2a --iteration_sync).

Data Validation Integration

When dataValidation is set to true, the flow performs a bit-exact verification of all received data at the end of every iteration.

• Highly effective for catching transient data corruption in complex A2A patterns.

• Validation occurs during the "post-iteration" management phase, outside of the timed performance interval.

Bandwidth Measurement

In bidirectional iteration-sync mode, bandwidth includes both TX and RX bytes. In unidirectional mode each
directed connection reports its own one-way bandwidth. Variable-A2A connections retain those per-connection
values, while the combined synchronized result reports aggregate BW as N/A.

Limitations

• Due to the heavy synchronization barrier, the measured "streaming" bandwidth will be lower than a standard continuous A2A test.

• A single scenario file cannot mix synchronization types. All tests must be with or without iteration-sync.

Peer Shifting

peerShifting runs an ALL_TO_ALL mesh as N-1 sequential rounds instead of all at once: in round p every endpoint i sends to endpoint (i+p) % N, so each endpoint has exactly one peer per round. Across the rounds every ordered endpoint pair is measured, giving a per-route bandwidth or latency figure for the whole fabric without full-mesh contention.

Supported in both JSON mode ("peerShifting": true) and CLI mode (--traffic_pattern all_to_all --peer_shifting).

"testNodes": [ {"hostName": "compute[01-08]", "deviceName": "mlx5_0"} ],
"trafficPattern": "ALL_TO_ALL",
"trafficDirection": "UNIDIR",
"peerShifting": true

This resolves 56 connections grouped into 7 rounds of 8. Each round is timed separately and the rounds run one after another, all reported under a single test. If a pair fails it is reported as failed and the remaining rounds still run. In the JSON output every connection carries its round in generalConfig.trafficPhaseId.

Hostname, Device Name, and CUDA Device ID Ranged Selection

To streamline configuration for multi-node and multi-device scenarios, DOCA Perftest supports bracket-based range expansion in both JSON and CLI modes. This allows you to define large-scale clusters concisely.

Supported Syntax

Expansion Logic

When ranges are defined for both hostnames and device names, the tool generates all possible combinations (Cartesian product).

For example:

• Input: hostName=host[1-2], deviceName=mlx5_[0-1]

• Result (4 Connections):

  1. host1 ↔ mlx5_0

  2. host1 ↔ mlx5_1

  3. host2 ↔ mlx5_0

  4. host2 ↔ mlx5_1

CUDA Device ID Ranged Selection

The cudaDeviceId field supports the same bracket-based range expansion syntax. When both deviceName and cudaDeviceId use ranges, values are paired by order (not as a Cartesian product).

Examples:

• deviceName = "mlx5_[0-1]" with cudaDeviceId = "[1,0]" creates the pairs: [mlx5_0, GPU1] and [mlx5_1, GPU0].

• deviceName = "mlx5_[0-3]" with cudaDeviceId = "[0,0,1,1]" creates the pairs: [mlx5_0, GPU0], [mlx5_1, GPU0], [mlx5_2, GPU1], [mlx5_3, GPU1].

Duplicate Connections

Duplicate (hostName, deviceName, cudaDeviceId (if exists)) connections are filtered out after pattern resolution.

Multiprocess (Cores)

DOCA Perftest can run synchronized multi-process tests, ensuring traffic starts simultaneously across all cores.

By default, it runs a single process on one automatically selected core.

Process and core selection:

Examples:

# Run on 3 synchronized processes (cores auto-selected)
doca_perftest -d mlx5_0 -n <server> -N 3

# Auto-select cores while excluding specific cores
doca_perftest -d mlx5_0 -n <server> -N 4 --exclude_cores 0,1

# Run on specific cores
doca_perftest -d mlx5_0 -n <server> -C 5
doca_perftest -d mlx5_0 -n <server> -C 5,7
doca_perftest -d mlx5_0 -n <server> -C 5-9

Host data buffers are placed on the NUMA node of the RDMA device that transfers them, whichever cores the test runs on, so that data transfers do not cross the interconnect. For a device that reports no node of its own, the node of the first selected core is used instead. The resolved node is reported as numaNode in the test configuration of the output.

Working with GPUs – Device Selection

DOCA Perftest can automatically select the most suitable GPU for each network device based on PCIe topology proximity. The ranking follows NVIDIA's nvidia-smi topo hierarchy: NV > PIX > PXB > PHB > NODE > SYS. It also load-balances by choosing the least-loaded GPU among those with the best available connection type.

This ensures that the GPU closest to the NIC is chosen, minimizing latency and maximizing throughput.

Although auto-selection is the default behavior, users can still manually specify a GPU device using the -G argument in CLI mode, or the "cudaDeviceId" field in JSON mode.

# Manually choose a specific GPU
doca_perftest -d mlx5_0 -n server-name -G 0

# Automatically select both GPU and memory type (recommended)
doca_perftest -d mlx5_0 -n server-name -M cuda

# Deprecated syntax (still supported, equivalent to cuda_auto_detect)
doca_perftest -d mlx5_0 -n server-name --cuda 0

Working with GPUs – Memory Types

RDMA operations can leverage GPU memory directly, bypassing CPU involvement for maximum throughput and minimal latency.

DOCA Perftest supports several CUDA memory modes optimized for different hardware and driver configurations.

Auto-Detection Mode (cuda_auto_detect)

Automatically selects the best available CUDA memory type in this order:

1. Data Direct

2. DMA-BUF

3. Peermem

This is the recommended mode for most users.

# Auto-detect best GPU memory type (recommended)
doca_perftest -d mlx5_0 -n server-name -M cuda -G 0

# With custom CUDA library path
doca_perftest -d mlx5_0 -n server-name -M cuda -G 0 --cuda_lib_path /usr/local/cuda-12/lib64

# Deprecated but equivalent syntax
doca_perftest -d mlx5_0 -n server-name --cuda 0

Standard CUDA Memory (cuda_peermem)

Traditional CUDA peer-memory allocation.

Supported on all CUDA-capable systems, though with slightly higher overhead compared to newer methods.

# Explicitly force peermem (bypasses auto-detect)
doca_perftest -d mlx5_0 -n server-name -M cuda_peermem -G 0

DMA-BUF Memory (cuda_dmabuf)

Uses the Linux DMA-BUF framework for zero-copy GPU–NIC transfers. Requires CUDA 11.7+ and kernel support.

doca_perftest -d mlx5_0 -n server-name -M cuda_dmabuf -G 0

Data Direct Memory (cuda_data_direct)

Most efficient GPU memory access method using direct PCIe mappings. Requires specific hardware and driver support; provides the lowest latency and highest throughput.

doca_perftest -d mlx5_0 -n server-name -M cuda_data_direct -G 0

Memory Types

Beyond GPU memory types, DOCA Perftest supports several memory allocation strategies for RDMA operations.

Host Memory (host)

Default mode using standard system RAM.

# Default host memory usage
doca_perftest -d mlx5_0 -n <server-name>

# Explicitly specify host memory
doca_perftest -d mlx5_0 -n <server-name> -M host

Null Memory Region (nullmr)

Does not allocate real memory; useful for ultra-low-latency synthetic tests.

# Null memory region for bandwidth testing
doca_perftest -d mlx5_0 -n <server-name> -M nullmr

Device Memory (device)

Allocates memory directly on the adapter hardware (limited by on-board capacity).

doca_perftest -d mlx5_0 -n <server-name> -M device

RDMA Drivers

Two RDMA driver backends are supported:

Atomic Verbs

RDMA atomic operations (FETCH_AND_ADD, COMPARE_AND_SWAP) are supported on RC queue pairs with the libibverbs driver (-r ibv).

Constraints:

Example:

doca_perftest -d mlx5_0 -n server-name -m lat -v atomic_fa -D 10
doca_perftest -d mlx5_0 -n server-name -m bw -v atomic_cs -D 10
doca_perftest -d mlx5_0 -n server-name -m bw --operations write:65536,atomic_fa:8,atomic_cs:8 -D 10

Auto-launching Remote Server

DOCA Perftest can automatically launch the remote server via SSH (CLI-only).

Requires passwordless SSH and identical versions on both sides.

Prerequisites:

• Available only in CLI mode (not relevant for JSON configuration).

• Requires passwordless SSH access to the server machine.

• Both client and server must have the same DOCA Perftest version.

# Auto-launch server (default)
doca_perftest -d mlx5_0 -n server-name

# Disable auto-launch
doca_perftest -d mlx5_0 -n server-name --launch_server disable

Running server detection:

The system checks if a server is already running on the target host. If detected, it connects to the existing server instead of launching a new one. This prevents multiple server instances and allows sharing servers between tests.

Server override examples:

# Server uses different device than client
doca_perftest -d mlx5_0 -n server-name --server_device mlx5_1

# Server uses different memory type
doca_perftest -d mlx5_0 -n server-name -M host --server_mem_type cuda_auto_detect

# Server runs on specific cores
doca_perftest -d mlx5_0 -n server-name -C 0-3 --server_cores 4-7

# Alternate server executable path
doca_perftest -d mlx5_0 -n server-name --server_exe /tmp/other_doca_perftest_version

# Different SSH username, supported by passwordless-ssh
doca_perftest -d mlx5_0 -n server-name --server_username testuser

QP Histogram

The QP histogram provides visibility into how work is distributed across multiple queue pairs during a test. This is useful for identifying load balancing issues, scheduling inefficiencies, or hardware limitations when using multiple QPs.

Enabling QP histogram:

Command Line Interface (CLI)

CLI execution is strictly restricted to one process (-N 1 is the default).

# Enable QP histogram with multiple queue pairs
doca_perftest -d mlx5_0 -n server-name -q 8 -H

# Shorthand flag
doca_perftest -d mlx5_0 -n server-name -q 8 --qp_histogram

JSON Configuration Mode

Supports multi-process execution.

"printQpHistogram": true

Example CLI output:

--------------------- QP WORK DISTRIBUTION ---------------------
Qp num 0:  ████████████████████████                     45.23 Gbit/sec  |  Relative deviation: -2.1%
Qp num 1:  █████████████████████████                    46.89 Gbit/sec  |  Relative deviation: 1.5%
Qp num 2:  ████████████████████████                     45.67 Gbit/sec  |  Relative deviation: -1.2%
Qp num 3:  █████████████████████████████                48.21 Gbit/sec  |  Relative deviation: 4.3%

JSON output:

The histogram data is nested within the resultsPerProcess array:

"connections": [
    {
        "resultsPerProcess": [
            {
                "qpHistogram": [
                    {
                        "qpIndex": 0,
                        "bwGbSec": 45.23,
                        "relativeDeviation": -2.1
                    },
                    {
                        "qpIndex": 1,
                        "bwGbSec": 46.89,
                        "relativeDeviation": 1.5
                    },
                    {
                        "qpIndex": 2,
                        "bwGbSec": 45.67,
                        "relativeDeviation": -1.2
                    },
                    {
                        "qpIndex": 3,
                        "bwGbSec": 48.21,
                        "relativeDeviation": 4.3
                    }
                ]
            }
        ]
    }
]

Start Packet Sequence Number

Start PSN controls the initial Packet Sequence Number for each Queue Pair (QP) at connection initialization. If unspecified, a random value is generated.

This feature is essential for debugging sequence-sensitive behavior, ensuring reproducibility, and interoperability testing.

Data Validation

Data validation verifies the integrity of RDMA traffic during bandwidth tests. When enabled, the requestor generates a deterministic payload for each message, and the responder compares the received data against the expected pattern.

To enable validation, set "dataValidation": true in JSON or use --validate in CLI mode.

Prerequisites and Constraints

• Test type: Supported only for bandwidth tests (latency testing is not supported).

• Supported modes:

  • Standard send verb tests.

  • Tests running in iteration-sync mode.

• Buffer configuration: rxDepth must be greater than or equal to txDepth.

• Warmup: Warmup must be explicitly disabled (-w 0).

• Enhanced Reliability (ER): If "ER Auto Mode" is set, it will be automatically disabled when validation is active.

Output and Reporting

When validation is enabled, the JSON output includes a validationResults section.

• Key metric: invalidDataSampleCount (the total number of messages that failed validation).

• Logging: Individual failure logs are capped at the first 5,000 invalid samples. Additional failures are counted in the metric but not logged individually.

If validation is disabled, this section is omitted entirely.

Enhanced Connection Establishment

ECE is an optional RDMA setup phase that aligns connection capabilities between the client and server before traffic begins.

When enabled, DOCA Perftest exchanges ECE parameters for each connection, leveraging the hardware-firmware negotiation that occurs during the Queue Pair (QP) transition from RESET to INIT.

High‑Level Flow

The ECE process ensures both sides agree on supported features before establishing the connection.

1. The client queries its local ECE capabilities and sends them to the server via the control channel.

2. The server applies the client's proposal, transitions its QP to INIT, and queries the device for the final accepted ECE configuration.

3. The server sends the finalized ECE configuration back to the client.

4. The client applies the finalized configuration, transitions its QP to INIT, and validates the negotiated result.

5. Standard QP data exchange and RTR/RTS transitions proceed as usual.

ECE Configuration

Limitations and Constraints

QP Hints

DOCA RDMA Verbs supports attaching opaque Congestion Control (CC) hints to Queue Pairs (QPs) for use by the Programmable Congestion Control (PCC) algorithm.

DOCA Perftest allows users to provide a binary hints file along with specific metadata (file size, vendor ID, and format ID). These parameters are passed directly to the PCC via the DOCA RDMA Verbs driver.

Configuring QP Hints

You can configure QP hints via CLI or JSON.

• CLI – Pass a comma-separated list containing the file path and metadata using the --cc_group_hints flag.

• JSON input – Add the ccGroupHints object to your test configuration:

"ccGroupHints": {
    "filePath": "/path/to/hints.bin",
    "fileSize": 1024,
    "vendorId": 1,
    "formatId": 1
}

TPH

PCIe optimization providing hints to CPUs for cache management and reduced memory-access latency.

Parameters:

Examples:

# Invalid: Core ID without memory type
doca_perftest -d mlx5_0 -n server-name --tph_core_id 0  # ERROR

# Invalid: Memory type without core ID
doca_perftest -d mlx5_0 -n server-name --tph_mem pm  # ERROR

# Valid: Both or neither
doca_perftest -d mlx5_0 -n server-name --ph 1  # OK (hints only)
doca_perftest -d mlx5_0 -n server-name --ph 1 --tph_core_id 0 --tph_mem pm  # OK (full config)

Warmup Configuration

By default, DOCA Perftest performs a time-based warmup before measuring results. Two warmup modes are available:

The two modes are mutually exclusive.

Constraints:

• Iteration-sync tests reject duration-based warmup; use --warmup_iterations instead.

• Latency tests reject iteration-based warmup; use -w instead.

• Data validation requires warmup to be explicitly disabled (-w 0).

End-margin

In multi-process or multi-node setups, processes may start at slightly different times. This means some processes finish measurement while others are still running, potentially causing the early finishers to stop sending traffic (which can affect the late starters' results).

The --end_margin option keeps traffic running for a specified number of seconds after measurement ends, ensuring all processes see consistent traffic throughout their measurement window.

Defaults:

• CLI single-process (no --traffic_pattern): 0 (disabled)

• CLI multi-process / --traffic_pattern / JSON mode: 2 seconds

End-margin applies to BW tests only and is not applicable for GPUNetIO. The reported bandwidth and iteration count reflect the measurement period only (end-margin traffic is excluded from results).

Live Bandwidth Reporting

The --report_interval (-R) flag enables periodic live bandwidth reporting during CLI bandwidth tests. When set, a scrolling log of throughput samples (Gb/s, Mpps) is printed at the configured interval.

# Print live BW every 2 seconds
doca_perftest -d mlx5_0 -n server-name -m bw -R 2

Default is 0 (disabled). Live reporting is supported in unidirectional, bidirectional, and
dual-unidirectional modes.

QP Transport Timeout

The --qp_timeout flag (CLI) or "qpTimeOut" field (JSON) configures the transport-level timeout for each Queue Pair. This controls how long the QP waits before retransmitting an unacknowledged packet.

doca_perftest -d mlx5_0 -n server-name --qp_timeout 18

Multi-verb Operations

The --operations flag (CLI) or "operations" field (JSON) enables posting multiple Work Requests per iteration, each with a different RDMA verb and message size. This is useful for benchmarking mixed-verb workloads.

# Post a Write of 64KB followed by a Send of 4KB per iteration
doca_perftest -d mlx5_0 -n server-name --operations write:65536,send:4096

Each WR in the group gets its own verb, message size, and memory region.

Automation and CI Integration

DOCA Perftest provides several features designed for integration with automation frameworks, CI/CD pipelines, and regression testing systems.

Performance Thresholds

Performance thresholds allow automated pass/fail validation of benchmark results. When a threshold is set, DOCA Perftest compares the measured results against the specified value and reports whether the test passed or failed.

If any test fails its threshold check, doca_perftest exits with return code 2 (instead of the normal 0). This enables CI systems to detect regressions by checking the process exit code.

Both thresholds default to 0 (disabled). The threshold and pass/fail status are included in the JSON output.

Examples:

# Bandwidth regression test: fail if below 90 Gbit/sec
doca_perftest -d mlx5_0 -n server-name -m bw --bw_threshold 90

# Latency regression test: fail if median exceeds 5 usec
doca_perftest -d mlx5_0 -n server-name -m lat --lat_threshold 5.0

Non-interactive Mode

The --non_interactive flag suppresses real-time progress indicators (spinners, progress bars, live table redraws). This is recommended when doca_perftest output is redirected to a file or pipe, or when a monitoring system reads stdout.

doca_perftest -d mlx5_0 -n server-name --non_interactive

Structured JSON Output to stdout

The --print_json_results flag (CLI) or "printJsonOutput": true field (JSON) prints the aggregated results summary to stdout in JSON format. This is the same schema used in the JSON output file.

This JSON output is treated as a stable API contract and will not change without a schema version bump. It is the recommended method for programmatic result consumption.

# CLI: print JSON results to stdout
doca_perftest -d mlx5_0 -n server-name --print_json_results

# JSON mode: add to scenario config
"printJsonOutput": true,
"outputFile": "results.json"

Connection Matrix Report

JSON scenarios and CLI --traffic_pattern runs print a directed connection matrix for each test and message size by default. Rows are source endpoints and columns are destination endpoints, labelled host:device, or host:device:gpu when the traffic runs from GPU memory. Bandwidth cells use red for lower values and green for higher values; latency uses the opposite scale. When a threshold is configured, cells use pass/fail colors instead of the heatmap.

A test with more than 16 endpoints prints its best and worst 10 connections instead of a matrix, since a wider matrix no longer fits a terminal. The JSON output retains all connection results either way.

Set "printConnectionMatrix": false at the scenario level to suppress the report for every test, on a single test to suppress only that one, or pass --disable_connection_matrix in a CLI multi-node run.

Running DOCA Perftest on BlueField-3 Devices

DOCA Perftest is capable of generating traffic from either the x86 host or the BlueField Arm cores, determined entirely by the input JSON configuration.

MPI Network Configuration

When launching doca_perftest from the server (regardless of whether the traffic originates from the x86 host or the BlueField), it is recommended to explicitly specify the MPI TCP network interface.

Add the subnet that connects the management server and the BlueField devices to the mpiTcpNetworkInterfaces field in your JSON input (e.g., "mpiTcpNetworkInterfaces": "10.7.8.0/24").

Traffic Originating from x86 Host (Server)

In this mode, traffic is generated by the x86 server. The RDMA device on the host (e.g., mlx5_0) performs DMA operations directly to/from host DRAM via PCIe.

Data path:

• Path: NIC ↔ PCIe ↔ Host Memory

• Bottlenecks: Performance is influenced by PCIe bandwidth and host CPU behavior, in addition to the network link and NIC capabilities.

JSON configuration:

• hostName: Set to the x86 server hostname.

• deviceName: Set to the RDMA device on the server (e.g., mlx5_0).

Traffic Originating from BlueField (Arm Cores)

In this mode, traffic is generated by the BlueField Arm cores, even if the test is launched from the x86 server. The RDMA device on the BlueField (e.g., p0, p1, mlx5_2, pf0vf0, depending on the operating mode) performs DMA operations to/from the BlueField's on-board DDR.

Data path:

• Path: NIC ↔ DPU DDR (No PCIe hop)

• Bottlenecks: Performance is typically limited by the network link, NIC, and DPU DDR bandwidth. The PCIe bus is not involved in the data path.

JSON Configuration:

• hostName: Set to the BlueField hostname.

• deviceName: Set to the RDMA device on the BlueField (e.g., p0, mlx5_2).

SLURM

DOCA Perftest integrates seamlessly with SLURM job schedulers, leveraging MPI for multi-node orchestration within SLURM allocations.

The following is a basic usage example with salloc:

1. Allocate nodes via SLURM (e.g., salloc -N8).

2. Update the JSON to include the allocated nodes. Simple bisection example:

"testNodes": [  {"hostName": "rack1-[01-04]", "deviceName": "mlx5_0"},
                {"hostName": "rack2-[05-08]", "deviceName": "mlx5_0"} ],
"trafficPattern": "BISECTION"

3. Run doca_perftest with the updated JSON:

doca_perftest -f <updated-json>

GPUNetIO Engine (GPU-initiated RDMA)

DOCA Perftest supports offloading the entire RDMA data-path to an NVIDIA GPU using DOCA GPUNetIO.

When the gpunetio engine is selected, WQE preparation, doorbell ringing, and CQ polling all execute inside CUDA kernels on the GPU. The host CPU is only involved during setup, teardown, and result collection.

This is useful for benchmarking RDMA as experienced by the GPU itself, which is the relevant data-path for GPU-centric workloads like distributed AI training.

Key points:

• Requires DOCA Verbs driver (-r dv) and CUDA memory (-M cuda)

• Supports Write verb only (-v write), both BW and Latency metrics

• Unidirectional traffic only (no bidirectional)

• Requires doca-gpunetio package, CUDA Toolkit, and a Volta+ GPU

Basic usage:

# CLI
doca_perftest -d mlx5_0 -n server-name -M cuda -G 0 -r dv --engine gpunetio

"engine": "gpunetio",
"memoryType": "cuda",
"cudaDeviceId": 0,
"rdmaDriver": "dv",
"verb": "write",
"metric": "bw"

For more information about DOCA GPUNetIO, refer to the DOCA GPUNetIO Programming Guide.

Last updated: September 01, 2026