Networking NVIDIA ConnectX NIC Triage Guide
Networking NVIDIA ConnectX NIC Triage Guide
Breadcrumbs

RoCE Transport Triage

Scope

Use this section when a RoCE (RDMA over Converged Ethernet) workload cannot establish RDMA connections, times out, disconnects, reports transport errors, or performs poorly while the adapter, PCIe connection, and Ethernet link are healthy.

Because RoCE carries RDMA traffic over Ethernet, check both endpoints and every switch on the path.

Quick Decision

What you see

Do this next

The Ethernet link is down or has physical errors.

Refer to Physical Link, Cable, Optic, FEC, and BER.

The RDMA device or RoCE mode is missing.

Refer to Firmware, Driver, and Operating-System Alignment.

A basic endpoint-to-endpoint RDMA test fails.

Refer to Validate the End-to-End Path.

A basic test works, but the workload fails.

Refer to RDMA, Application, and NCCL Workload Triage.

The issue follows one rail, NIC, or interface.

Check rail and interface selection and the host-to-switch mapping. Refer to Timeout, Disconnect, or Performance Loss.

The issue follows a traffic class or switch path.

Check traffic classification, congestion, and switch-path evidence. Refer to Validate the End-to-End Path.

Record the Intended Transport Design

Before you change configuration, record the intended design:

  • RoCE version, RDMA device, port, and interface or rail

  • Endpoint IP address, VLAN or overlay, and routing path

  • Maximum transmission unit (MTU) and traffic classification

  • Whether the transport is designed to be lossless or lossy

  • Congestion-control design, including whether it uses Priority Flow Control (PFC) and Explicit Congestion Notification (ECN)

  • Affected scope: one endpoint pair, one rail, one switch path, or the wider fabric

Do not assume that every RoCE deployment uses PFC. Confirm the documented design for the platform first.

Validate the End-to-End Path

Compare both endpoints and the switch path between them for the following:

  • RDMA device and RoCE mode

  • IP reachability and intended route

  • MTU

  • Traffic class, Differentiated Services Code Point (DSCP) and priority mapping, and VLAN or overlay membership

  • PFC and ECN settings, only where the documented design requires them

  • Congestion-control configuration and counter behavior

  • Consistent switch configuration across the affected path or plane

Check every hop. A mismatch in traffic classification or congestion policy anywhere on the path can cause the failure even when both NICs are configured correctly.

Timeout, Disconnect, or Performance Loss

  1. Resolve any physical-link errors first.

  2. Collect good-state and bad-state host bundles at both endpoints.

  3. Collect switch counters and congestion evidence for the same time window.

  4. Compare the failing path with a healthy path or rail.

  5. On a multi-NIC system, confirm that the workload uses the intended interface and rail.

Escalation Criteria

Contact NVIDIA Enterprise Support or your OEM/reseller support provider, depending on your support agreement, if any of the following conditions applies:

  • The end-to-end configuration is confirmed, but a basic RDMA test still fails.

  • The failure follows a switch path, rail, traffic class, or UDP source port used for path selection, instead of one endpoint.

  • Counters show congestion, loss, pause, or marking behavior that does not match the documented design.

  • The platform uses a specialized transport design that needs a product-specific owner.

Attachments

  • Host bundles from both endpoints

  • RDMA device and RoCE mode

  • Interface, rail, and host-to-switch mapping

  • IP address, VLAN, and route details

  • Expected MTU and traffic classification

  • Approved transport design

  • Relevant switch configuration and counters

  • Incident timing

  • Result of a controlled endpoint-to-endpoint test

For the full escalation package, see Escalation Package and Handoff.

Last updated: