E-W Fabric Health

Verifies that East-West fabric health — RoCE adaptive retransmissions, Congestion Notification Packet (CNP) handling, ECN marking, and TX/RX pause-frame activity — is at the expected idle baseline. East-West pressure is a frequent cause of training step-time variance, and idle-baseline values become the "is this normal?" reference once workloads run.

Persona Use Cases

Validation Procedure

Step 1: Confirm Idle Adaptive Retransmissions

1. In the user-tenant, open the Traffic and Congestion dashboard.

2. Inspect Congestion Control (RoCE):

   • Adaptive Retransmissions: visualizes hwRoceAdpRetrans counts. This is the primary metric for identifying packet loss in the fabric. At idle, this should be at or near zero.

   • CNP Handled: Congestion Notification Packets handled at the NIC level (hwRpCnpHandled). At idle, this should be near zero.

Step 2: Confirm Idle TX ECN Marking

1. Open the Professor-X Dashboard.

2. Inspect RoCE Performance & Congestion:

   • TX ECN Marked (ifTcTxEcnMarkedTc) across all tiers: the panel visualizes which tier is under congestion pressure. At idle, all tiers should show near-zero ECN marking.

   • Adaptive Retransmissions and Adaptive Retransmissions Timeouts (hwRoceAdpRetrans, hwRoceAdpRetransTo): idle baseline.

Step 3: Confirm Idle Pause-frame Activity

1. On Professor-X Dashboard > RoCE Performance & Congestion > TX/RX Pauses, confirm pause-frame counts are quiet at idle. Frequent pauses at the leaf-to-NIC level often indicate a "slow receiver" (compute node) issue.

2. On Traffic and Congestion > Pause Frames, confirm both TX and RX Pauses per role (leaf, spine, super-spine) are at idle baseline.

Step 4: Confirm Discards Are Absent

1. On Traffic and Congestion > Discards, inspect Discards - NIC (ifInDiscards) at the host level. At idle, this should be zero.

Step 5: Cross-check on AI Cluster Health

1. Open AI Cluster Health.

2. Inspect Metric Performance Anomalies > Adaptive Retransmits (RoCE) and Interface Error Rate. Both should report no anomaly activity at idle.

Step 6: Capture the Baseline

Idle-baseline values for Adaptive Retransmissions, CNP Handled, ECN Marked, TX/RX Pauses, and NIC Discards are the most-referenced numbers in later step-time analysis. Archive screenshots and, where useful, the actual numeric values per tier.

Reference Dashboards and Sections

• Traffic and Congestion: Fabric Throughput, Link & Interface Health, Congestion Control (RoCE), Discards.

• Professor-X Dashboard: Performance & Fabric Bandwidth, RoCE Performance & Congestion.

• AI Cluster Health: Metric Performance Anomalies.

• Background reading: Pause Frame Anomaly Correlation illustrates how an idle-baseline drift triggers downstream incidents.

Source Metric Mapping

Pass Criteria

• [ ] Adaptive Retransmissions (hwRoceAdpRetrans) is at or near zero at idle.

• [ ] Adaptive Retransmissions Timeouts (hwRoceAdpRetransTo) is at or near zero at idle.

• [ ] CNP Handled (hwRpCnpHandled) is at or near zero at idle.

• [ ] TX ECN Marked is near zero across leaf, spine, and super-spine tiers.

• [ ] TX and RX Pauses are quiet across all tiers; in particular, no sustained leaf-to-NIC RX pauses.

• [ ] Discards - NIC (ifInDiscards) is zero at idle.

• [ ] No Adaptive Retransmits (RoCE) anomalies are firing on AI Cluster Health Metric Performance Anomalies.

• [ ] Idle-baseline numeric values per tier are captured and archived.

Common Pitfalls

• Pre-existing PFC misconfiguration manifests as steady non-zero pauses. A small, flat pause rate at idle suggests a Priority Flow Control configuration issue (wrong DSCP-to-priority mapping, wrong queue assignments), not real congestion. Distinguish flat-baseline from spiky-onset patterns.

• CNP Handled at zero with high ECN Marked is a red flag. ECN marks should be acted on; if leaves are marking but NICs are not handling CNPs, the configuration is broken end to end.

• Leaf-to-NIC RX pauses at idle. This often indicates a "slow receiver": a NIC or host that is unable to drain ingress fast enough. Escalate to the compute-ops team for investigation; the underlying cause is rarely fabric-side.

• Idle baseline captured during a still-ramping cluster. If install-time activity (NTP sync storms, telemetry agent backfill) is still in progress, the baseline will overstate real idle. Allow 15–30 minutes of post-install settling before capturing.

Last updated: October 21, 2026