Validate East-West Fabric Congestion

Verifies that the East-West fabric's congestion signals — RoCE adaptive retransmissions, Congestion Notification Packet (CNP) handling, ECN marking, and TX/RX pause-frame activity — are at their expected idle baseline. Congestion in the East-West fabric is a frequent cause of variance in training step time, so the values you capture at idle become the reference for judging whether readings are normal once workloads run.

This validation area is part of the AI Factory Operational Readiness Review. Before you begin, complete the steps in Prepare for Readiness Validation.

Persona Use Cases

Validation Procedure

Step 1: Confirm Idle Adaptive Retransmissions

1. In the user-tenant, open the Traffic and Congestion dashboard.

2. Inspect Congestion Control (RoCE):

   • Adaptive Retransmissions: visualizes hwRoceAdpRetrans counts. This is the primary metric for identifying packet loss in the fabric. At idle, this should be at or near zero.

   • CNP Handled: Congestion Notification Packets handled at the NIC level (hwRpCnpHandled). At idle, this should be near zero.

Step 2: Confirm Idle TX ECN Marking

1. Open the Professor-X Dashboard.

2. Inspect RoCE Performance and Congestion:

   • TX ECN Marked (ifTcTxEcnMarkedTc) across all tiers: the panel visualizes which tier is under congestion pressure. At idle, all tiers should show near-zero ECN marking.

   • Adaptive Retransmissions and Adaptive Retransmissions Timeouts (hwRoceAdpRetrans, hwRoceAdpRetransTo): idle baseline.

Step 3: Confirm Idle Pause-frame Activity

1. On Professor-X Dashboard > RoCE Performance and Congestion > TX/RX Pauses, confirm pause-frame counts are quiet at idle. Frequent pauses at the leaf-to-NIC level often indicate a "slow receiver" (compute node) issue.

2. On Traffic and Congestion > Pause Frames, confirm both TX and RX Pauses per role (leaf, spine, super-spine) are at idle baseline.

Step 4: Confirm Discards Are Absent

1. On Traffic and Congestion > Discards, inspect Discards - NIC (ifInDiscards) at the host level. At idle, this should be zero.

Step 5: Cross-check on AI Cluster Health

1. Open AI Cluster Health.

2. Inspect Metric Performance Anomalies > Adaptive Retransmits (RoCE) and Interface Error Rate. Both should report no anomaly activity at idle.

Step 6: Capture the Baseline

Idle-baseline values for Adaptive Retransmissions, CNP Handled, ECN Marked, TX/RX Pauses, and NIC Discards are the most-referenced numbers in later step-time analysis. Archive screenshots and, where useful, the actual numeric values per tier.

Reference Dashboards and Sections

• Traffic and Congestion: Fabric Throughput, Link & Interface Health, Congestion Control (RoCE), Discards.

• Professor-X Dashboard: Performance and Fabric Bandwidth, RoCE Performance and Congestion.

• AI Cluster Health: Metric Performance Anomalies.

• Background reading: Pause Frame Anomaly Correlation illustrates how an idle-baseline drift triggers downstream incidents.

Source Metric Mapping

Pass Criteria

• [ ] Adaptive Retransmissions (hwRoceAdpRetrans) is at or near zero at idle.

• [ ] Adaptive Retransmissions Timeouts (hwRoceAdpRetransTo) is at or near zero at idle.

• [ ] CNP Handled (hwRpCnpHandled) is at or near zero at idle.

• [ ] TX ECN Marked is near zero across leaf, spine, and super-spine tiers.

• [ ] TX and RX Pauses are quiet across all tiers; in particular, no sustained leaf-to-NIC RX pauses.

• [ ] Discards - NIC (ifInDiscards) is zero at idle.

• [ ] No Adaptive Retransmits (RoCE) anomalies are firing on AI Cluster Health Metric Performance Anomalies.

• [ ] Idle-baseline numeric values per tier are captured and archived.

Common Pitfalls

• Pre-existing PFC misconfiguration manifests as steady non-zero pauses. A small, flat pause rate at idle suggests a Priority Flow Control configuration issue (wrong DSCP-to-priority mapping, wrong queue assignments), not real congestion. Distinguish flat-baseline from spiky-onset patterns.

• CNP Handled at zero with high ECN Marked is a red flag. ECN marks should be acted on; if leaves are marking but NICs are not handling CNPs, the configuration is broken end to end.

• Leaf-to-NIC RX pauses at idle. This often indicates a "slow receiver": a NIC or host that is unable to drain ingress fast enough. Escalate to the compute-ops team for investigation; the underlying cause is rarely fabric-side.

• Idle baseline captured during a still-ramping cluster. If install-time activity (NTP sync storms, telemetry agent backfill) is still in progress, the baseline will overstate real idle. Allow 15–30 minutes of post-install settling before capturing.

Last updated: October 30, 2026