ECMP and Multiplane Balance

Verifies that flowing traffic distributes evenly across ECMP next-hops and across the parallel "rails" or planes of a multi-plane topology. The current preconfigured dashboard set does not include a dedicated ECMP-balance dashboard. The ORR is necessarily a limited check: at idle there is little traffic to imbalance, so the goal is to confirm uniformly low traffic across all members and capture the baseline that workload-time validation will compare against.

Persona Use Cases

Validation Procedure

Step 1: Validate Idle Uniformity on the Performance Dashboard

1. In the user-tenant, open the Performance Dashboard (imported from the Reference Configuration Package under minerva-views/user-tenant/jobs-and-performance/).

2. Inspect the per-tier RX/TX panels:

   • RX By Interface - Leaf (priorityGroup: 1)

   • RX By Interface - Spine - Downlinks (toward leaves)

   • RX By Interface - Spine - Uplinks (toward super-spines)

   • RX By Interface - SuperSpine

   • TX By Interface - Leaf (trafficClass: 3)

   • TX By Interface - Spine - Downlinks

   • TX By Interface - Spine - Uplinks

   • TX By Interface - SuperSpine

3. At idle, every panel should show uniformly low traffic across all member interfaces. Any single member showing significantly higher activity than its peers is an ORR finding: it usually points to an unscheduled flow or to a misconfigured static route.

Step 2: Confirm Max BW Per Level Idle Baseline

1. On the Professor-X Dashboard > Performance & Fabric Bandwidth > Max BW Per Level, check peak ifOutOctetRate across the leaf, spine, and super-spine tiers.

2. At idle, all four tiers should show low and balanced peak bandwidth.

Step 3: Confirm SU Activity Is Uniform

1. On Professor-X Dashboard > SU Activity, check the maximum NIC-to-leaf bandwidth for each pod and scalable unit (SU).

2. At idle, every SU should show similar low activity. An SU showing visible activity at idle is unusual and worth investigating.

Step 4: Document the Baseline for Future Comparison

1. Capture screenshots of the Performance Dashboard per-tier RX/TX panels and the Professor-X Dashboard Max BW Per Level panel.

2. Note in the baseline archive that ECMP balance cannot be fully validated until representative workload traffic exists. This is a post-Day 0 dependency.

Step 5: (If Feasible) Run a Synthetic ECMP Probe

If the change-control process and team capacity allow, you can use a synthetic east-west traffic generator (for example, iperf3 or an NCCL AllReduce micro-benchmark across two SUs) to validate ECMP distribution before workloads land:

1. Run a short-duration multi-flow probe across two pods.

2. Confirm RX/TX per spine member during the probe is uniform within approximately 10% across same-tier members.

3. Document probe parameters and outcome in the baseline archive.

This step is optional during the ORR but is the strongest pre-workload validation available.

Reference Dashboards and Sections

• Performance Dashboard (imported from the Reference Configuration Package under minerva-views/user-tenant/jobs-and-performance/).

• Professor-X Dashboard: Performance & Fabric Bandwidth.

• Traffic and Congestion: Fabric Throughput for a Max BW Per Level cross-check.

Pass Criteria

• [ ] All Performance Dashboard per-tier RX/TX panels show uniformly low traffic across member interfaces at idle.

• [ ] Professor-X Dashboard Max BW Per Level shows low, balanced peak bandwidth across leaf, spine, and super-spine.

• [ ] Professor-X Dashboard SU Activity shows similar low activity across all SUs.

• [ ] No single ECMP member is significantly above its peers in any tier at idle.

• [ ] Idle-baseline screenshots archived with an explicit "ECMP balance is workload-dependent and will be re-validated once workloads run" note.

• [ ] (Optional) Synthetic ECMP probe results documented if performed.

Common Pitfalls

• "Uniformly low" is hard to verify visually at the line-chart level. Panels at idle look like flat lines near zero, and small relative imbalances are only visible at log scale or via a dedicated standard-deviation-across-members panel — neither of which exists in the preconfigured set. Make clear to the team that this is a coarse ORR check.

• A single-member spike at idle. This often traces to an out-of-band management flow (firmware push, telemetry replay, NTP sync storm). Identify the source rather than assuming ECMP imbalance.

• Multi-plane configuration mismatch. If the cluster is supposed to have N parallel rails or planes, confirm that Topology (Pods, SUs) shows N planes' worth of links per node before drawing conclusions from this page. ECMP balance against a topologically incomplete fabric is meaningless.

• Hash polarization at multi-stage spine. In multi-tier ECMP, identical hash functions at each stage can cause flows to pile onto the same path despite per-stage balance. This is a workload-time finding, not an ORR finding.

Last updated: October 21, 2026