Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

NVLink Interfaces

Persona Use Cases

The NVLink Interfaces Dashboard provides a granular, interface-level view of the health and error rates across the entire AI cluster's NVLink fabric, covering both servers and NVSwitches.

Persona

Primary Use Case

Infrastructure/DC Ops

Identifies specific failing physical links by monitoring Raw Bit Error Rates (BER) and Local Link Integrity Errors to schedule cable or tray replacements.

System Administrators/SREs

Monitors Effective Errors (Post-FEC) to identify interfaces where the hardware can no longer correct bit-flips, indicating imminent failure.

AI/ML Engineers

Correlates job performance jitter with PLR Retransmissions to determine if the network physical layer is causing NCCL stalls.

Hardware Engineering

Analyzes Symbol Error Rates and Raw BER trends to validate hardware revisions or firmware stability across the fleet.

Dashboard Overview

The NVLink Interfaces Dashboard aggregates health metrics and supervised logs across the entire AI cluster's NVLink interfaces. By default, it presents data for the last 24 hours. While the NVLink Health Dashboard focuses on high-level node status, this view provides the telemetry needed to isolate issues to specific Device:Interface pairs.

Note: This dashboard is organized into logical tiers: Summary Health, Supervised Log Anomalies, and Deep Metric Telemetry (separated by server and switch).

Health Issues (Summary Tiles)

The top section provides real-time counts of interfaces currently experiencing critical degradations.

  • Critical Interface Errors: Total count of interfaces with active uncorrectable error states.

  • High BER Links: Number of links exceeding the established baseline for Raw Bit Error Rates.

  • Link Integrity Alerts: Interfaces reporting physical layer instability or frequent resets.

Log Errors

This section uses Supervised Learning (Collective Learning) to surface high-impact events extracted from system logs.

  • NVLink Collective Learning Logs HM: A heatmap visualizing supervised log events across the cluster. It maps specific error categories against deviceName to identify areas of fabric instability.

  • Supervised Categories: Tracks specific events like NVLink link-down, port-disablement, and Fabric Manager initialization failures.

Metric Errors (Interface Telemetry)

This section provides heatmaps and trends for specific physical layer metrics.

  • Description: Displays all error bits detected at the Physical Layer (PHY) per interface.

  • Significance: Includes both correctable (by FEC) and uncorrectable bits. A high Raw BER is often the first sign of a failing optic or cable before the user experiences job impact.

  • Key Widgets:

    • NVLink Raw Bit Error Rate - by Device:Interface

    • NVLink Server Raw Bit Error Rate

  • Description: Tracks errors related to the stability of the physical connection between peers.

  • Impact: Frequent integrity errors usually lead to link flaps or automatic port downing by the driver.

Symbol Error Rate

  • Description: Monitors Layer 2 stability. High symbol error rates indicate that the link is struggling to maintain synchronization, even if individual bit errors are being corrected.

Effective Errors (Post-FEC)

  • Description: This widget specifically filters for uncorrectableErrors > 0.

  • Impact: Critical. These are errors that Forward Error Correction (FEC) could not fix. Any data passing through these links is corrupted, likely triggering an XID or job crash.

  • Description: Monitors the Physical Layer Retransmit (PLR) mechanism.

  • Significance: PLR allows the link to retry a failed transmission at the hardware level. While this prevents a crash, high retransmission rates introduce latency that degrades AI training Step Time.

Last updated: