Networking NVIDIA ConnectX NIC Triage Guide
Networking NVIDIA ConnectX NIC Triage Guide
Breadcrumbs

Physical Link, Cable, Optic, FEC, and BER

Use this section when a network port is down, flaps, trains at an unexpected speed, shows growing Forward Error Correction (FEC) or Bit Error Rate (BER) counters, or reports a cable, optic, or module warning.

Start here only after you confirm that the adapter is visible and its PCIe connection is healthy.

Quick Decision

Symptom

Action

The port is down or never comes up.

Refer to Establish the Physical Picture, and then to Collect Link Evidence from Both Ends.

The port flaps or drops under load.

Refer to Collect Link Evidence from Both Ends, and Collect Link Evidence from Both Ends with the incident time.

BER, FEC, or lane errors increase.

Refer to Narrow the Fault Domain. Identify the affected lane or port, and compare one component at a time.

The port is stable and error-free.

Refer to InfiniBand Fabric Triage, RoCE Transport Triage, RDMA, Application, and NCCL Workload Triage, or Performance, Topology, and GPUDirect Triage.

Establish the Physical Picture

For the affected port and its peer, record the following:

  • Adapter, port, and peer switch-port mapping

  • Cable or optic part number and serial number

  • Expected protocol, speed, lane count, and FEC mode

  • Current link state, negotiated speed, and module state

  • Incident time, and whether the problem follows a cable, optic, port, or peer in a controlled comparison

Collect link diagnostics from both endpoints. For the ConnectX endpoint, follow the mlxlink Utility documentation for the installed MFT release. For the switch endpoint, use the equivalent diagnostics documented for that switch platform.

For a flap or intermittent error, collect the following:

  • Good-State evidence while the link is healthy

  • Bad-State evidence immediately after the link drops or the counters grow

Compare the following:

  • Link state, speed, lane count, and FEC mode

  • Module readiness, temperature, power, and fault state

  • Per-lane faults and BER and FEC counters

  • Link-down reason and link-down count

  • Cable or optic identity and firmware, where reported

Narrow the Fault Domain

Collect link evidence from both ends before you touch hardware. Then make one controlled change at a time:

  1. Reseat and inspect the existing cable or optic according to site procedure.

  2. Replace it with a known-good compatible cable or optic.

  3. If practical, move the link to a known-good port or peer path.

  4. After each change, record which component the symptom followed.

Do not change several components at once. A recovery after a change identifies a suspect component, and the comparison result confirms the root cause.

Escalation Criteria

Contact NVIDIA Enterprise Support or your OEM/reseller support provider, as specified in your support agreement, when any of the following conditions applies. Include diagnostic evidence from both ends of the affected link.

  • The link stays down after you confirm the expected configuration and peer mapping.

  • BER or FEC errors persist or grow after a controlled component comparison.

  • Both ends show the same symptom, but it does not follow the cable or optic.

  • The failure is intermittent, and the Good-State and Bad-State evidence shows a state change.

  • The evidence points to a firmware, protocol-negotiation, or fabric-level cause instead of a local physical component.

Attachments

  • Host bundle

  • mlxlink output from both ends

  • Port mapping

  • Cable or optic identity

  • Expected and actual link settings

  • Incident timing

  • Result of each controlled comparison

For the full escalation package, see Escalation Package and Handoff.

External Reference

mlxlink utility

Last updated: