Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

PHY Errors

Verifies that the physical layer is healthy across the cluster's NVLink and Ethernet fabric: bit error rates are within baseline, FEC is correcting errors as expected, and Physical Layer Retransmit (PLR) activity is absent or minimal at idle. PHY-layer issues are leading indicators of failing optics or cables; catching them during the ORR prevents NCCL stalls and step-time degradation later.

Persona Use Cases

Persona

Primary Use Case

Infrastructure / DC Ops

Identifies specific failing physical links by monitoring Raw Bit Error Rate (BER) and Local Link Integrity Errors to schedule cable or tray replacements.

System Administrators / SREs

Monitors Effective Errors (Post-FEC) to identify interfaces where FEC can no longer correct bit flips, signaling imminent failure.

AI/ML Engineers

Correlates job performance jitter with PLR Retransmissions to determine if the network physical layer is causing NCCL stalls.

Hardware Engineering

Analyzes Symbol Error Rates and Raw BER trends to validate hardware revisions or firmware stability across the fleet.

Validation Procedure

  1. In the user-tenant, open the NVLink Interfaces dashboard.

  2. Inspect Health Issues (Summary Tiles):

    • Critical Interface Errors: total count of interfaces with active uncorrectable error states. Should be 0.

    • High BER Links: links exceeding the established Raw BER baseline. Should be 0.

    • Link Integrity Alerts: interfaces reporting physical-layer instability or frequent resets. Should be 0.

  1. On NVLink Interfaces > Metric Errors (Interface Telemetry):

    • NVLink Raw Errors – Bit errors per bit (Pre-Correction): inspect NVLink Raw Bit Error Rate – by Device:Interface and NVLink Server Raw Bit Error Rate. Both should be at or near the established baseline.

    • Local Link Integrity Errors: no widespread non-zero values.

    • Symbol Error Rate: monitors Layer-2 stability. High counts indicate the link is struggling to maintain synchronization.

    • Effective Errors (Post-FEC): filters for uncorrectableErrors > 0. Any non-zero count here means data is corrupted past FEC, which will trigger XID errors or job crashes.

    • NVLink Server PLR Retransmissions: high retransmission rates introduce latency that degrades AI training step time.

Step 3: Validate Switch-side PHY Uncorrectable Errors

  1. Open AI Cluster Health.

  2. Inspect Health Issues (Summary Tiles) > Switch PHY Uncorrectable Errors and Switch Interfaces with PHY Uncorrectable Errors. Both should be 0.

  3. Drill into Metric Errors > Switch PHY Uncorrectable Errors heatmap. The heatmap visualizes specific interfaces incrementing the Effective Errors (Post-FEC) counters on a switch.

Step 4: Validate NIC Phy Symbol Errors

  1. On AI Cluster Health > Metric Errors > NIC Phy Symbol Errors, confirm the heatmap is empty across the idle window.

  2. Cross-check Switch/NIC Symbol Errors on the same dashboard for layer2SymbolErrors activity.

Step 5: Validate Collective Learning Surfaces

  1. On AI Cluster Health, inspect the Log Errors section for Network Link Disconnections or Instability and any other PHY-related categories from the Collective Learning library. Heatmap hot spots indicate whether an issue is systemic or localized.

Reference Dashboards and Sections

Pass Criteria

  • [ ] Critical Interface Errors, High BER Links, and Link Integrity Alerts summary tiles on NVLink Interfaces all read 0.

  • [ ] NVLink Raw Bit Error Rate – by Device:Interface is at or below the established baseline across all interfaces.

  • [ ] Effective Errors (Post-FEC) is 0 everywhere: no uncorrectableErrors > 0.

  • [ ] NVLink Server PLR Retransmissions is at or near zero at idle.

  • [ ] Switch PHY Uncorrectable Errors summary tile on AI Cluster Health reads 0.

  • [ ] Switch PHY Uncorrectable Errors, NIC Phy Symbol Errors, and Switch/NIC Symbol Errors heatmaps on AI Cluster Health are empty.

  • [ ] No PHY-related category is active on the Log Errors Collective Learning heatmap.

  • [ ] Idle-baseline screenshots of NVLink Interfaces and AI Cluster Health Metric Errors archived.

Common Pitfalls

  • A small flat baseline on Raw BER is normal. Optics have a non-zero error floor; what matters is that Raw BER is flat and FEC corrects it (Effective Errors stays at zero). A trending or spiking Raw BER is the concerning signal.

  • Symbol errors localized to a single rack. Often points to a single bad cable run or transceiver; the NVLink Nodes/Switches with Phy Symbol Errors detail tables provide deviceName and rack to localize.

  • Effective Errors > 0 even at very low rates. Treat any uncorrectable error as actionable. FEC was supposed to fix it; if it didn't, the data is corrupt and downstream impact (XIDs, job crashes) is only a matter of time.

  • Server-side PLR Retransmissions at a non-zero rate. PLR allows the link to retry at the hardware level, which prevents a crash but introduces latency. This often shows up later as job step-time variance. Capture the idle-baseline rate so it can be compared during workload runs.

Last updated: