Performance, Topology, and GPU-Direct Triage | NVIDIA ConnectX NICs Triage Guide

Use this section when connectivity is healthy, but throughput is low, latency is high, bandwidth is uneven across participants, or a collective slows down as the job grows.

Before you start, check adapter visibility, PCIe state, physical link health, and the InfiniBand or RoCE transport path.

Quick Decision

Symptom	Action
One port or rail has link errors or retraining.	Refer to the Collect Link Evidence from Both Ends
PCIe speed, width, or locality differs from the expected value.	Refer to Device Visibility and PCIe.
Basic RDMA bandwidth is poor.	Refer to InfiniBand Fabric Triage or RoCE Transport Triage.
Basic RDMA is healthy, but collective performance is poor.	Refer to GPU-to-NIC Locality and Rail Balance, and compare workload placement, topology, rail balance, and collective behavior.
One node, GPU, NIC, or rail is slower than the others.	Treat it as an imbalance. Refer to Isolate the Bottleneck, and compare it with a known-good peer.

Establish the Performance Baseline

Record the expected and observed results:

Test or workload, message size, iteration count, and concurrency
Node count, rank count, GPU count, and rail count
Aggregate result, and the result for each node, GPU, and rail
Expected topology and GPU-to-NIC locality
PCIe and NUMA placement
Fabric or Ethernet path, switch tier, and traffic class
Software stack and runtime configuration

Record the result for each node, GPU, and rail as well as the aggregate. An aggregate number can hide one slow rail or participant.

Isolate the Bottleneck

Use the applicable Ethernet/RoCE or InfiniBand diagnostics to assess traffic rates, errors, discards, and congestion or flow-control indicators where available. Refer to Physical Link, InfiniBand Fabric Triage, or RoCE Transport Triage for the relevant collection guidance.

Compare counter changes over the same test interval on both endpoints and, where available, the intervening switch ports. Compare the affected path with a known-good path using the same workload. Interpret counters according to the device and software documentation; counter names and availability vary by protocol, platform, and release.

Use this evidence together with controlled performance tests to narrow the bottleneck to the sending endpoint, receiving endpoint, or network path. Then follow these steps:

Compare a controlled endpoint-to-endpoint result with the workload result.
Compare one node, GPU, NIC, and rail at a time with a healthy peer.
Check whether the low result follows the GPU, NIC, PCIe or NUMA placement, rail, switch path, or job placement.
Reduce the workload by nodes, ranks, GPUs, or rails to find the first scale at which the regression appears.
For intermittent or run-to-run variation, collect good-state and bad-state host bundles.

Don't tune collective, routing, or congestion settings until the counters and a controlled comparison identify the bottleneck layer. Aim for a root cause that NVIDIA can fix in the default product configuration. A tuning change at one site does not help other customers who hit the same issue.

Measurement Tools for Customers

Use these tools to measure performance and collect evidence before you change any tuning settings.

Tool	Use
NVIDIA SOS (`doca-sosreport`)	Host bundle with host, adapter, driver, and counter evidence, on a host with DOCA-Host when the required NIC plugins complete
`mlxlink`	NIC, PCIe, cable, and port health (not for continuous monitoring)
`ib_write_bw` (Perftest)	Controlled point-to-point RDMA bandwidth
`ib_write_lat` (Perftest)	Point-to-point latency and tail-latency comparison
`rdma statistic`	RDMA retries and protocol-error counters during a test
NCCL tests and logs	Collective performance and imbalance; see RDMA, Application, and NCCL Workload Triage

Use the same test parameters on the failing path and the known-good path. Record the message size, concurrency, endpoints, device or rail, and time window with every result.

After identifying the bottleneck, mlnx_tune, where available in the installed NVIDIA networking software package, can assist with Linux host tuning. Use a profile only when it is appropriate for the platform and workload and recommended by the platform documentation or support provider. Preserve the baseline configuration and compare performance before and after the change. Tuning does not replace evidence collection or root-cause investigation.

For an example of using these point-to-point tools and reading rdma statistic output in a NIC investigation, see NCCL Networking Troubleshooting.

GPU-to-NIC Locality and Rail Balance

For GPUDirect or collective workloads, confirm that the expected GPU, NIC, and PCIe and NUMA relationships are in place.

Compare the following:

GPU-to-NIC locality and assigned NIC or rail
PCIe generation and width
Traffic and bandwidth for each rail
Rank placement and process affinity
Whether the slow result follows the hardware path or the workload placement

If one rail carries less traffic or finishes later than the others, record the mapping and Refer to the matching transport section or to Device Visibility and PCIe.

Escalation Criteria

Contact NVIDIA Enterprise Support or your OEM/reseller support provider, depending on your support agreement, if any of the following conditions applies:

The baseline and controlled comparisons show a reproducible imbalance.
The bottleneck follows topology, locality, rail assignment, or job placement.
A basic network test is healthy, but collective scaling degrades.
The evidence points to product-specific GPUDirect, collective, or topology behavior.

Attachments

Exact test or workload configuration
Expected and observed results
Results for each node, GPU, and rail
Topology, and GPU, NIC, PCIe, and NUMA mapping
Host bundles
Transport evidence
Smallest reproducer that shows the regression

For the full escalation package, see Escalation Package and Handoff.

Last updated: October 08, 2026