Networking NVIDIA ConnectX NIC Triage Guide
Networking NVIDIA ConnectX NIC Triage Guide
Breadcrumbs

RDMA, Application, and NCCL Workload Triage

Use this section when a controlled endpoint-to-endpoint RDMA test passes, but an RDMA application, collective, or NVIDIA Collective Communications Library (NCCL) workload fails, hangs, times out, or performs poorly.

Quick Decision

What you see

Do this next

The adapter, PCIe, or physical link is unhealthy.

Refer to Device Visibility and PCIe or Physical Link, Cable, Optic, FEC, and BER, depending on the symptom.

A basic endpoint-to-endpoint RDMA test fails.

Refer to InfiniBand Fabric Triage or RoCE Transport Triage.

Basic RDMA works, but one workload fails.

Refer to Capture the Workload Evidence, and isolate the affected process, rank, node, or rail.

The failure follows a specific NIC or network path.

Refer to the relevant section: Physical Link, Cable, Optic, FEC, and BER, InfiniBand Fabric Triage, or RoCE Transport Triage. Include the NIC, rail, or path mapping.

The failure is a collective hang, imbalance, or low bandwidth.

Refer to Performance, Topology, and GPUDirect Triage.

Capture the Workload Evidence

Before you change environment variables, network policy, or application settings, record the following:

  • Exact command, container or package version, and workload configuration

  • Start time, timeout, error text, and reproduction rate

  • Participating nodes, process or rank mapping, and affected interfaces or rails

  • Whether a smaller job, a single node, or a controlled endpoint-to-endpoint test succeeds

  • Good-state and bad-state host bundles, if the failure is intermittent

  • Transport and switch evidence for the same time window

Isolate the Failure Domain

  1. Run a controlled endpoint-to-endpoint RDMA test on the affected path.

  2. If the test fails, Refer to InfiniBand Fabric Triage or RoCE Transport Triage.

  3. If the test passes, reduce the workload: use fewer nodes, fewer ranks, or one rail at a time.

  4. Determine whether the failure follows a node, rank, NIC, rail, topology placement, or workload version.

  5. Compare the failing case with a known-good workload run, changing one setting at a time.

Don't change several workload and network settings together. Treat a workaround as a lead until a controlled comparison confirms what changed.

NCCL First Checks

For an NCCL collective issue, collect the exact environment, the rank-to-node mapping, the incident time, and a separate NCCL log for each rank.

Purpose

NCCL Setting

Standard focused reproduction

NCCL_DEBUG=INFO

Focus on setup and network transport

NCCL_DEBUG_SUBSYS=INIT,NET

Deeper diagnosis of the smallest reproducer

NCCL_DEBUG=DEBUG or NCCL_DEBUG=TRACE

Preserve logs per rank

NCCL_DEBUG_FILE=<unique file per rank>

  1. Run the focused reproduction with NCCL_DEBUG=INFO and preserve output from every participating rank.

  2. Add NCCL_DEBUG_SUBSYS=INIT,NET when the investigation is about initialization or network transport.

  3. If INFO does not identify the failing stage, repeat the smallest reproducer with NCCL_DEBUG=DEBUG or NCCL_DEBUG=TRACE.

  4. Pair the NCCL logs with endpoint snapshots and switch evidence from the same time window.

Use high verbosity only for a scoped reproduction. TRACE can generate a large volume of logs and should not be enabled broadly or indefinitely.

Interpret the Result

Finding

Next step

NCCL identifies a specific NIC, rail, or path.

Refer to the matching transport or physical-link section.

NCCL shows a setup, topology, or locality inconsistency.

Compare node, rank, GPU, and NIC placement with a known-good run.

A minimal NCCL test fails.

Keep the test output, and continue with step 3 of Isolate the Failure Domain.

A minimal NCCL test passes, but the application fails.

Compare the application configuration and communication pattern with the working test.

The failure appears only at scale.

Reduce by nodes, ranks, or rails to find the first scale that fails.

Escalation Criteria

Contact NVIDIA Enterprise Support or your OEM/reseller support provider, depending on your support agreement, if any of the following conditions applies:

  • Basic RDMA connectivity is healthy, but the workload failure still reproduces.

  • The failure follows a rank, process placement, workload version, or application configuration.

  • The collective fails only at scale or only with a specific topology.

Attachments

  • Exact workload command and configuration

  • NCCL log for each rank

  • Node, rank, GPU, NIC, interface, and rail mapping

  • Incident timing

  • Host bundles

  • Controlled RDMA test result

  • Minimal NCCL test result

  • Known-good comparison

  • Relevant transport or switch evidence

For the full escalation package, see Escalation Package and Handoff.

External Reference

NCCL NIC-level diagnostics

Last updated: