InfiniBand Fabric Triage

Use this section when the adapter and physical link are healthy but an InfiniBand endpoint cannot use the fabric. For example, the port stays in INIT, cannot reach the fabric or some of its peers, or the problem affects many nodes.

The Subnet Manager (SM) is the InfiniBand service that discovers the fabric and assigns the addresses and routes that devices need to communicate. Every active InfiniBand subnet needs an SM. NVIDIA Unified Fabric Manager (UFM) can host and manage the SM. The SM can also run as OpenSM on a host or on a managed switch.

Quick Decision

Collect the Fabric-Manager Package

Use the collection that matches the deployment.

A UFM system dump is the preferred UFM collection. It includes UFM logs, configuration, database, system configuration, and reports. It does not replace the host bundle, or a switch support bundle when you suspect a switch or link.

For a fabric-wide or topology-related case, collect ibdiagnet output from the same fabric and time window as the SM or UFM evidence. For UFM-managed fabrics, include the UFM diagnostic data and the ibdiagnet snapshot corresponding to the incident, when available. Avoid duplicate collection when the snapshot contains the required evidence. If the fabric is not managed by UFM, or the incident snapshot is unavailable, collect an ibdiagnet report using the documentation for the installed release

Confirm the Fabric State

Record the following:

• Adapter and port identity, globally unique identifier (GUID), and expected fabric or rail

• Local link state and peer switch-port mapping

• Fabric state as reported by the SM or UFM

• Assigned local identifier (LID) and partition membership, where applicable

• Scope: one port, one host, one rail, some peers, or the wider fabric

A port can be physically up while fabric initialization or reachability is incomplete, so check the SM or UFM view as well as the link state.

Endpoint in INIT or Missing from the Fabric

If the link is physically up but the endpoint stays in INIT or isn't visible in the fabric, follow these steps:

1. Preserve the host bundle and the fabric-manager package before you change configuration.

2. Confirm that the active SM manages the intended InfiniBand subnet.

3. Review the SM or UFM event log for the time the endpoint joined or failed to join.

4. Check the endpoint, port, and peer mapping against the intended topology.

5. Compare the affected endpoint with a healthy endpoint in the same role.

Reachability or Partition Problem

If the port is active but cannot communicate as expected, check the following:

• The expected peer set and traffic scope

• Partition membership and fabric policy

• The affected rail or path compared with a healthy one

• Whether the issue follows the endpoint, rail, switch path, or workload

• ibdiagnet output, for fabric-wide or topology-related cases

Escalation Criteria

Contact NVIDIA Enterprise Support or your OEM/reseller support provider, as specified in your support agreement, when any of the following conditions applies:

• The SM or UFM view conflicts with the physical-link evidence.

• The endpoint stays in INIT after you confirm the local link and peer mapping.

• The issue affects multiple nodes, rails, or paths.

• The issue is isolated to partitioning, routing, or fabric policy.

• Local evidence is clean, but the fabric-manager package or ibdiagnet output shows a broader condition.

Attachments

• Host bundle

• Adapter and switch-port mapping

• Expected topology or rail

• Fabric-manager package for the deployment

• ibdiagnet output

• Switch support bundles, when a switch or inter-switch link is involved

For the full escalation package, see Escalation Package and Handoff.

Last updated: October 08, 2026