Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

NVLink Health

Verifies that the high-speed GPU-to-GPU and GPU-to-NVSwitch interconnect fabric is healthy at idle. NVLink instability is a frequent silent cause of training job failures: XID 145 typically manifests as NCCL or CUDA timeouts rather than as obvious failures. Confirm a clean NVLink baseline before the cluster takes work.

Persona Use Cases

Persona

Primary Use Case

Infrastructure / DC Ops

Identifies physical-layer issues like failing cables or optics by monitoring Physical Symbol Errors and Link Down events.

System Administrators / SREs

Correlates XID 149.4 and XID 149.37 events to isolate fabric-manager issues or peer-to-peer connection resets.

AI/ML Engineers

Checks XID 145 counts and Bandwidth Loss Ratios to predict whether NVLink will introduce jitter into training jobs.

Validation Procedure

Step 1: Confirm Health Summary Tiles Are Clean

  1. In the user-tenant, open the NVLink Health Dashboard.

  2. Inspect Health Issues (Summary Tiles). At idle:

    • Nodes with XID 149.4 should be 0 (signal-integrity-related link resets).

    • Nodes with Link Down Events should be 0 (no node should have two or more link transitions within any 1-hour window over the past 24 hours).

    • Nodes with Phy Symbol Errors should be 0 or trending toward zero.

  3. Drill into any non-zero tile by clicking through to its detail table: the table includes deviceName, rack, jobId, and phySymbolErrors > 0 columns.

  1. Inspect NVLink Recovery Telemetry:

    • Physical Layer Symbol Errors (phy_symbol_errors): high counts indicate aging hardware or poor cable seating.

    • NVLink Nodes/Switches with Phy Symbol Errors: detail tables; should be empty.

    • Bandwidth Loss Ratio: should be at or near zero. Any non-zero ratio at idle is a leading indicator of training step-time degradation.

  1. Inspect Link Down & Connection Events:

    • XID 149.37 (Peer Connection): common during node reboots or Fabric Manager transitions, but should be quiet in steady state.

    • XID 149.4 (Signal Integrity): non-zero here usually requires hardware inspection.

    • Unintentional Link Downs table: switches with two or more link transitions within any 1-hour window over the past 24 hours.

Step 4: Validate XID 145 Events

  1. Inspect XID 145 Events. XID 145 is the canonical signal for NVLink instability, packet loss, and source timeouts.

  2. The Flat View of nodes throwing XID 145 should be empty. If a node already shows deviceCustomState = drain or futr, it has been sidelined by the scheduler: confirm with the workload-ops team that the sidelining is intentional.

Step 5: Cross-check on AI Cluster Health

  1. On AI Cluster Health, confirm the GPU Critical Errors summary already validated in GPU Health has not changed.

  2. The Log Errors section should not be highlighting XID 149.4 or XID 145 in its categorized events.

Reference Dashboards and Sections

Pass Criteria

  • [ ] Nodes with XID 149.4 summary tile reads 0.

  • [ ] Nodes with Link Down Events summary tile reads 0.

  • [ ] Nodes with Phy Symbol Errors summary tile reads 0 or near zero with no trend.

  • [ ] Bandwidth Loss Ratio is at or near zero across the idle window.

  • [ ] XID 149.37 event count is zero in the validation window.

  • [ ] XID 145 Flat View has no nodes listed (or any listed nodes are explained by an intentional scheduler drain).

  • [ ] No NVSwitch appears in the Unintentional Link Downs table.

  • [ ] Idle-baseline screenshot archived.

Common Pitfalls

  • Fresh-boot artifacts. A recently-rebooted node may show non-zero XID 149.37 counts that age out of the 24-hour window. If the cluster was rebooted during install, allow the events to fall outside the time filter before signing off.

  • Fabric Manager transitions during install. XID 149.37 is often seen during node reboots or Fabric Manager transitions (see NVLink Health Dashboard > Link Down & Connection Events). Distinguish operational events from real instability by correlating to install-window timestamps.

  • Symbol errors trending up at idle. A non-zero rate of symbol errors that is trending (not flat) is more concerning than a small flat baseline: it suggests an optic or cable that is degrading rather than a known-imperfect link.

  • drain/futr in deviceCustomState. A node already in drain or futr state will not appear in normal dashboards; check the scheduler-side state and the node's exclusion reason before concluding the NVLink fabric is clean.

Last updated: