Networking NVIDIA ConnectX NIC Triage Guide
Networking NVIDIA ConnectX NIC Triage Guide
Breadcrumbs

Common Intake and Evidence Collection

Follow this section first for every ConnectX NIC incident, before you classify it as a PCIe, physical-link, driver, InfiniBand, RoCE, RDMA, workload, or performance problem.

The goal is to preserve evidence and identify the fault domain before anyone reboots, resets, replaces hardware, or changes configuration.

Key Terms

Term

Meaning

Host Bundle

The diagnostic archive that doca-sosreport produces on a host. Every section in this guide uses it as the baseline evidence.

Good State

The last known healthy state of the adapter and path.

Bad State

The state immediately after the symptom appears.

Known-Good Comparison

A working sibling port, alternate cable or optic, comparable host, or previous supported software stack.

Safety and Preservation Rules

  1. Record the time and the first observed symptom.

  2. Do not reboot, reset the adapter, reflash firmware, reseat a module, or change configuration until you collect the host bundle, unless you must restore service first.

  3. For an intermittent incident, collect a good-state host bundle and a bad-state host bundle whenever you can.

  4. Collect evidence from both ends of a link. A report from one end cannot distinguish a local adapter fault from a cable, optic, peer, or configuration fault.

  5. Compare with a known-good sibling port, alternate cable or optic, comparable host, or previous supported software stack.

Incident Intake

Record the following before you collect logs.

Field

Required information

Symptom

Exact error text, user impact, affected application, and whether the condition is an outage, an intermittent failure, or performance degradation

Time

First occurrence, most recent occurrence, and whether the issue coincides with a maintenance window or deployment

Scope

One port, one adapter, one host, a subset of nodes, or the whole fabric

Reproduction

Exact steps or workload command, reproduction rate, and whether a minimal test reproduces the issue

Change history

Firmware, driver, operating system or kernel, BIOS, cable or optic, switch, topology, power, or configuration changes

Known-good reference

Known-good host, port, or path; previously stable version; or clean run after a power cycle

Topology

Host, adapter PCIe address, NIC port, switch port, peer endpoint, and protocol

Host Bundle Collection

If you can reach the host, collect the host bundle before any reboot.

Default Action: Collect the Host Bundle

On a host with the NVIDIA DOCA-Host package, use NVIDIA/doca-sosreport to collect the host bundle. NVIDIA DOCA is NVIDIA's software platform for networking and infrastructure services, and DOCA-Host is its host software package.

For full instructions, see Collecting DOCA Logs for NVIDIA Inspection. Attach the archive to the support case.

The host bundle contains system status, installed packages, adapter identity, firmware and configuration queries, NVIDIA Firmware Tools (MFT) data, driver and network information, RDMA and counter data, and local cable diagnostics. It also records whether each collection step succeeded.

Before you escalate, open sos_logs/sos.log and confirm that these plugins ran without errors:

  • mellanox_firmware

  • mlx5_core

  • devlink_health

  • pci

  • networking

  • rdma, infiniband, or roce, depending on the deployment

If a plugin failed or was disabled, record the gap in the case. Do not run unrelated commands to fill it.

If DOCA-Host is not installed or the required collection plugins cannot run, use NVIDIA SysInfo Snapshot, where supported, following its documented collection instructions. Preserve any partial host bundle and collection logs.

In the support case, record the affected host, collection time, tool and version, failed or unavailable plugin or command, exact error, and evidence that could not be collected. Attach the available archive and logs. If neither collection tool can run, contact your support provider for targeted collection instructions. Do not change the driver stack solely to collect diagnostics.

For an intermittent failure, follow these steps:

  1. Collect a Good-State Host bundle while the adapter and path are healthy.

  2. Reproduce the failure.

  3. Collect a Bad-State Host bundle immediately after the failure.

  4. Keep both bundles, and name each one with the host, the state, and a timestamp.

Review the bundle status before escalating. If required evidence is missing because a collection component failed or was disabled, record that gap.

Add Targeted Collection

Situation

Targeted Addition

UFM-managed InfiniBand fabric

UFM logs

Fabric-wide InfiniBand diagnosis

ibdiagnet output

GPU or GPUDirect workload

On each affected GPU host, run sudo nvidia-bug-report.sh and attach the resulting nvidia-bug-report.log.gz to the support case, unless equivalent GPU evidence is already included in the host bundle. Collect before reboot or reset whenever possible. See NVIDIA GPU log-collection guidance.

If the Adapter is not Visible

If the host cannot see the adapter, preserve the available host and platform logs before you reboot or reset the system. Then, refer to Device Visibility and PCIe.

Add Evidence the Host Bundle does not Contain

Do not collect information that's already in the host bundle. Add only the following evidence.

Evidence

Contents

Peer-side link evidence

mlxlink or equivalent port data from the far end of the link

Switch-side context

Switch model, software version, port mapping, and event history

Fabric-manager context

Subnet Manager (SM) or NVIDIA Unified Fabric Manager (UFM) evidence for InfiniBand incidents

Workload context

Exact workload command, timestamp, and affected rank or node

Known-good comparison

Results from a sibling port, alternate cable or optic, comparable host, or previous supported stack

Good-State and Bad-State Comparison

For every intermittent failure, compare good-state and bad-state evidence, unless the fault occurs immediately after power-on..

Term

Meaning

Good State

The last known healthy state of the adapter and path.

Bad State

The state immediately after the symptom appears.

Compare the following:

  • Adapter visibility and PCIe link state

  • Firmware, driver, and configuration inventory

  • Port state, speed, FEC mode, and module state

  • Link and error counters

  • Host events and fabric events

  • Workload results

Branch Selection

Observation

Start with

Adapter is missing, resets, or has a degraded PCIe link

Device Visibility and PCIe

Port is down or flapping, FEC or BER counters grow, or a module reports a warning

Physical Link, Cable, Optic, FEC, and BER

Issue started after a firmware, driver, operating system, or BIOS change

Firmware, Driver, and Operating-System Alignment

InfiniBand port does not reach Active, or the SM reports a topology error

InfiniBand Fabric Triage

RoCE RDMA fails while the IP path works

RoCE Transport Triage

RDMA queue-pair, memory registration, or application-only failure

RDMA, Application, and NCCL Workload Triage

Low bandwidth, imbalance, or GPU collective issue

Performance, Topology, and GPUDirect Triage

Escalation

If the selected section does not resolve the issue, prepare the package described in Escalation Package and Handoff.

External Reference

NVIDIA/doca-sosreport

Last updated: