Networking NVIDIA ConnectX NIC Triage Guide

Device Visibility and PCIe

Use this section when the adapter is missing from the host, resets repeatedly, appears only after a reboot, or runs at an unexpected PCIe speed or width.

If the adapter is visible and PCIe is healthy, but the port has a link, RoCE, InfiniBand, or workload problem, use the matching section instead.

Quick Decision

What you see

Do this next

The host can't see the adapter.

Preserve available logs. Do not reset or reflash yet. Refer to Adapter Absent from PCIe.

The adapter resets or disappears and then returns.

Collect Good-State and Bad-State host bundles if possible. Refer to Adapter Absent from PCIe.

The adapter is visible, but PCIe runs below the expected speed or width.

If the condition is intermittent, collect Good-State and Bad-State host bundles. Refer to PCIe Degraded.

The adapter is visible and PCIe is normal.

Refer to Physical Link, Cable, Optic, FEC, and BER, InfiniBand Fabric Triage, RoCE Transport Triage, RDMA, Application, and NCCL Workload Triage, or Performance, Topology, and GPUDirect Triage.

Adapter Missing or Unstable

Confirm the Symptom

Confirm that the adapter is missing from the host PCIe inventory, and not only from one application or network interface view. [TBD: add the command to check the PCIe inventory.]

Record the following:

  • Time of first observation

  • Host and expected adapter and port

  • Most recent Known-Good state

  • Recent firmware, driver, BIOS, power, or hardware changes

  • Any host reset, fatal event, or platform-management event

Preserve Evidence Before Recovery

Before you reboot the host or change hardware, follow these steps:

  1. If the host still responds, collect a host bundle. Keep the complete bundle even if the NIC firmware or MFT commands in it fail.

  2. Preserve host and platform logs for the incident window.

  3. Record whether the adapter returns after a reboot, power cycle, or reseat.

  4. If you can run a controlled comparison, record whether the condition follows the adapter, slot, riser, or host.

If the adapter returns after a reboot, the failure is intermittent. The reboot doesn't identify the root cause.

Adapter Absent from PCIe

Use this procedure when the host is running but the adapter no longer appears in the host PCIe inventory. In-band NIC diagnostics can't run in this state.

  1. Collect a host bundle before recovery.

  2. Record the expected adapter location, the first-observed time, recent changes, the last Known-Good state, and the NIC commands that failed.

  3. If the platform supports an approved out-of-band (OOB) path, use it to collect device data. MTUSB access is one option, and it requires hands-on access to the system to connect the MTUSB device.

  4. If no OOB path is available, escalate with the host bundle, incident timing, recent changes, Known-Good state, and recovery sequence. Do not run in-band diagnostics while the adapter is unavailable.

PCIe Degraded

Compare Expected and Actual PCIe State

Compare the negotiated PCIe speed and width with the platform specification and with a Known-Good sibling adapter or an earlier Good-State host bundle. [TBD: add the command that shows negotiated PCIe speed and width.]

A degraded PCIe link reduces throughput and can make a healthy port look like it has a network problem. Resolve the PCIe condition before you investigate the link, RoCE, or workload tuning.

Compare Good and Bad States

For intermittent degradation, compare Good-State and Bad-State host bundles for the following:

  • Adapter presence and PCIe address

  • Negotiated speed and width

  • Host and kernel events around the transition

  • Firmware, driver, and BIOS inventory

  • Topology and NUMA placement

  • Whether the condition changes after a controlled adapter, slot, riser, or host comparison

Escalation Criteria

Escalate to [TBD: name the device and PCIe escalation team or queue] when any of the following is true:

  • The adapter is still missing after standard recovery.

  • The same adapter repeatedly degrades, resets, or disappears.

  • The problem follows the adapter in a controlled comparison.

  • The PCIe state changes without a matching configuration change.

  • The incident does not reproduce, but the Good-State and Bad-State evidence shows a state change.

Attachments

  • Host bundles

  • Incident timing

  • Expected and actual PCIe state

  • Topology

  • Recent changes

  • Results of any controlled comparison

For the full escalation package, see Escalation Package and Handoff.

Last updated: