Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

Validate NVLink Health

Verifies that the high-speed GPU-to-GPU and GPU-to-NVSwitch interconnect fabric is healthy at idle. NVLink instability is a frequent silent cause of training job failures: XID 145 typically manifests as NCCL or CUDA timeouts rather than as obvious failures. Confirm a clean NVLink baseline before the cluster takes work.

This validation area is part of the AI Factory Operational Readiness Review. Before you begin, complete the steps in Prepare for Readiness Validation.

Persona Use Cases

Persona

Primary Use Case

Infrastructure / DC Ops

Identifies physical-layer issues like failing cables or optics by monitoring Physical Symbol Errors and Link Down events.

System Administrators / SREs

Correlates XID 149.4 and XID 149.37 events to isolate fabric-manager issues or peer-to-peer connection resets.

AI/ML Engineers

Checks XID 145 counts and Bandwidth Loss Ratios to predict whether NVLink will introduce jitter into training jobs.

Validation Procedure

Step 1: Confirm Health Summary Tiles Are Clean

  1. In the user-tenant, open the NVLink Health Dashboard.

  2. Inspect Health Issues (Summary Tiles). At idle:

    • Nodes with XID 149.4 should be 0 (signal-integrity-related link resets).

    • Nodes with Link Down Events should be 0 (no node should have two or more link transitions within any 1-hour window over the past 24 hours).

    • Nodes with Phy Symbol Errors should be 0 or trending toward zero.

  3. Drill into any non-zero tile by clicking through to its detail table: the table includes deviceName, rack, jobId, and phySymbolErrors > 0 columns.

  1. Inspect NVLink Recovery Telemetry:

    • Physical Layer Symbol Errors (phy_symbol_errors): high counts indicate aging hardware or poor cable seating.

    • NVLink Nodes/Switches with Phy Symbol Errors: detail tables; should be empty.

    • Bandwidth Loss Ratio: should be at or near zero. Any non-zero ratio at idle is a leading indicator of training step-time degradation.

  1. Inspect Link Down and Connection Events:

    • XID 149.37 (Peer Connection): common during node reboots or Fabric Manager transitions, but should be quiet in steady state.

    • XID 149.4 (Signal Integrity): non-zero here usually requires hardware inspection.

    • Unintentional Link Downs table: switches with two or more link transitions within any 1-hour window over the past 24 hours.

Step 4: Validate XID 145 Events

  1. Inspect XID 145 Events. XID 145 is the canonical signal for NVLink instability, packet loss, and source timeouts.

  2. The Flat View of nodes throwing XID 145 should be empty. If a node already shows deviceCustomState = drain or futr, it has been sidelined by the scheduler: confirm with the workload-ops team that the sidelining is intentional.

Step 5: Cross-check on AI Cluster Health

  1. On AI Cluster Health, confirm the GPU Critical Errors summary has not changed since you checked it in Validate GPU Health.

  2. The Log Errors section should not be highlighting XID 149.4 or XID 145 in its categorized events.

Reference Dashboards and Sections

Pass Criteria

  • [ ] Nodes with XID 149.4 summary tile reads 0.

  • [ ] Nodes with Link Down Events summary tile reads 0.

  • [ ] Nodes with Phy Symbol Errors summary tile reads 0 or near zero with no trend.

  • [ ] Bandwidth Loss Ratio is at or near zero across the idle window.

  • [ ] XID 149.37 event count is zero in the validation window.

  • [ ] XID 145 Flat View has no nodes listed (or any listed nodes are explained by an intentional scheduler drain).

  • [ ] No NVSwitch appears in the Unintentional Link Downs table.

  • [ ] Idle-baseline screenshot archived.

Common Pitfalls

  • XID 149.37 events from reboots or Fabric Manager transitions during install. XID 149.37 is often seen during node reboots or Fabric Manager transitions (see Link Down and Connection Events in NVLink Health Dashboard). These events age out of the 24-hour window. If the cluster was rebooted during install, correlate the events with install-window timestamps, or let them fall outside the time filter before signing off.

  • Symbol errors trending up at idle. A non-zero rate of symbol errors that is trending (not flat) is more concerning than a small flat baseline: it suggests an optic or cable that is degrading rather than a known-imperfect link.

  • drain/futr in deviceCustomState. A node already in drain or futr state will not appear in normal dashboards; check the scheduler-side state and the node's exclusion reason before concluding the NVLink fabric is clean.

Last updated: