Use this section when the adapter is missing from the host, resets repeatedly, appears only after a reboot, or runs at an unexpected PCIe speed or width.
If the adapter is visible and PCIe is healthy, but the port has a link, RoCE, InfiniBand, or workload problem, use the matching section instead.
Quick Decision
|
What you see |
Do this next |
|---|---|
|
The host can't see the adapter. |
Preserve available logs. Do not reset or reflash yet. Refer to Adapter Absent from PCIe. |
|
The adapter resets or disappears and then returns. |
Collect Good-State and Bad-State host bundles if possible. Refer to Adapter Absent from PCIe. |
|
The adapter is visible, but PCIe runs below the expected speed or width. |
If the condition is intermittent, collect Good-State and Bad-State host bundles. Refer to PCIe Degraded. |
|
The adapter is visible and PCIe is normal. |
Refer to Physical Link, Cable, Optic, FEC, and BER, InfiniBand Fabric Triage, RoCE Transport Triage, RDMA, Application, and NCCL Workload Triage, or Performance, Topology, and GPUDirect Triage. |
Adapter Missing or Unstable
Confirm the Symptom
Confirm that the adapter is missing from the host PCIe inventory, and not only from one application or network interface view. [TBD: add the command to check the PCIe inventory.]
Record the following:
-
Time of first observation
-
Host and expected adapter and port
-
Most recent Known-Good state
-
Recent firmware, driver, BIOS, power, or hardware changes
-
Any host reset, fatal event, or platform-management event
Preserve Evidence Before Recovery
Before you reboot the host or change hardware, follow these steps:
-
If the host still responds, collect a host bundle. Keep the complete bundle even if the NIC firmware or MFT commands in it fail.
-
Preserve host and platform logs for the incident window.
-
Record whether the adapter returns after a reboot, power cycle, or reseat.
-
If you can run a controlled comparison, record whether the condition follows the adapter, slot, riser, or host.
If the adapter returns after a reboot, the failure is intermittent. The reboot doesn't identify the root cause.
Adapter Absent from PCIe
Use this procedure when the host is running but the adapter no longer appears in the host PCIe inventory. In-band NIC diagnostics can't run in this state.
-
Collect a host bundle before recovery.
-
Record the expected adapter location, the first-observed time, recent changes, the last Known-Good state, and the NIC commands that failed.
-
If the platform supports an approved out-of-band (OOB) path, use it to collect device data. MTUSB access is one option, and it requires hands-on access to the system to connect the MTUSB device.
-
If no OOB path is available, escalate with the host bundle, incident timing, recent changes, Known-Good state, and recovery sequence. Do not run in-band diagnostics while the adapter is unavailable.
PCIe Degraded
Compare Expected and Actual PCIe State
Compare the negotiated PCIe speed and width with the platform specification and with a Known-Good sibling adapter or an earlier Good-State host bundle. [TBD: add the command that shows negotiated PCIe speed and width.]
A degraded PCIe link reduces throughput and can make a healthy port look like it has a network problem. Resolve the PCIe condition before you investigate the link, RoCE, or workload tuning.
Compare Good and Bad States
For intermittent degradation, compare Good-State and Bad-State host bundles for the following:
-
Adapter presence and PCIe address
-
Negotiated speed and width
-
Host and kernel events around the transition
-
Firmware, driver, and BIOS inventory
-
Topology and NUMA placement
-
Whether the condition changes after a controlled adapter, slot, riser, or host comparison
Escalation Criteria
Escalate to [TBD: name the device and PCIe escalation team or queue] when any of the following is true:
-
The adapter is still missing after standard recovery.
-
The same adapter repeatedly degrades, resets, or disappears.
-
The problem follows the adapter in a controlled comparison.
-
The PCIe state changes without a matching configuration change.
-
The incident does not reproduce, but the Good-State and Bad-State evidence shows a state change.
Attachments
-
Host bundles
-
Incident timing
-
Expected and actual PCIe state
-
Topology
-
Recent changes
-
Results of any controlled comparison
For the full escalation package, see Escalation Package and Handoff.
Last updated: