The Multi-Plane Overview suite provides operational visibility into P0–P3 fabric planes for AI factory clusters using Spectrum-X Ethernet fabric. Use these dashboards during active training jobs to compare network planes, isolate congestion, identify packet drops, quantify queue pressure, and locate traffic imbalances.
Persona Use Cases
|
Persona |
Primary Use Case |
|---|---|
|
Network Engineers |
Compare traffic, ECN, and drop counters across P0–P3 to determine whether a symptom is plane-specific or fabric-wide. |
|
Performance Engineers |
Identify which plane is carrying disproportionate load or showing elevated queue pressure during active jobs. |
|
HPC/Slurm Admins |
Correlate plane-level drop or ECN symptoms with job performance regressions to determine whether the network is the limiting factor. |
|
Cluster Administrators |
Triage fabric health at a glance and navigate to the appropriate drill-down dashboard before opening per-device views. |
Dashboard Overview
The suite installs six dashboards and 26 supporting trend views. All dashboards share a common filter set (plane P0–P3) and use raw_v1 telemetry for switch interface, queue, traffic, ECN, pause, and packet-drop counters. The NIC RoCE Metrics dashboard additionally uses system_v1 Amber component telemetry.
Multi-Plane Overview is the triage entry point. The remaining five dashboards are drill-downs that open directly from the "What to Do Next" section on the overview. See Import Dashboards.
|
Dashboard |
Purpose |
|---|---|
|
Multi-Plane Overview |
Triage — compare P0–P3 across traffic, ECN, drops, and queue symptoms |
|
Multi-Plane Overview – Traffic Discrepancy |
Load balance — locate where planes diverge in TX+RX traffic |
|
Multi-Plane Overview – ECN RoCE Congestion |
ECN and pause — congestion-feedback signals at switch and link level |
|
Multi-Plane Overview – Packet Drop Root Cause |
Packet loss — classify drop type and locate dropping devices |
|
Multi-Plane Overview – Queue Analytics |
Queue depth — buffer occupancy, egress wait, and PG pressure |
|
Multi-Plane Overview – NIC RoCE Metrics |
NIC congestion control — server-side CC symptoms by plane and host |
Multi-Plane Overview
The main overview dashboard is the triage entry point for the suite. It compares P0–P3 across traffic, ECN, drop, and queue signals so the next drill-down dashboard is obvious without having to open device-level views first. The "What to Do Next" section contains direct links to the five drill-down dashboards.
Plane Balance
Plane Balance | Traffic Heatmap P0–P3: A heatmap with planes (P0–P3) on one axis and fabric role on the other. The value is summed TX+RX octets (planeTraffic) across the selected 24-hour window. Planes carrying similar traffic appear uniformly shaded; imbalance shows as a bright or dark column and points to routing, placement, or pathing skew. This is the first panel to check when one job or pod appears under-utilized.
Triage Signals
These widgets sit below the balance heatmap and provide a side-by-side comparison of the four key symptom categories.
Multi-Plane Score: A compact heatmap summary of P0–P3 health derived from traffic, ECN, and packet-drop inputs (multiPlaneScore). The score is a triage signal, not a root-cause indicator — use it only to decide which drill-down dashboard to open next.
Traffic by Plane: A bar chart of total plane traffic (ifTcTxOctet + ifPgRxOctet) summed over time, with one series per active plane. Use it as the base load-balance signal.
ECN by Plane: A bar chart of switch-side ECN marks (ifTcTxEcnMarkedTc) by plane. ECN can appear before drops; elevated ECN on one plane while others are quiet indicates class-local congestion.
Drops by Plane: A bar chart of total HLD packet-drop counters (drops) by plane. If drops dominate the signal, navigate to the Packet Drop Root Cause dashboard.
Worst Plane: A table ranking all four planes by traffic, ECN, and drop totals. Use it to decide which plane needs attention first.
Plane / Pod / SU Focus: A table that translates plane-level symptoms into fabric location by grouping plane, groupPod, and su. Columns include trafficGB, ifTcTxEcnMarkedTc, and drops. Use it after identifying the worst plane to narrow the scope to a specific pod or Scalable Unit.
Multi-Plane Overview – Traffic Discrepancy
Use this dashboard when one plane carries too much or too little traffic relative to its peers. It shows whether the imbalance is cluster-wide, pod-local, rail-specific, or role-dependent, and lists the largest interface contributors to the skew.
Traffic Balance
Traffic by Plane: A bar chart of total TX+RX traffic (ifTcTxOctet + ifPgRxOctet) by plane over time. This is the primary load-balance signal; planes with similar workloads should track closely.
Skew Localization
These heatmaps isolate where traffic imbalance starts. Each uses total TX+RX traffic (traffic) as the measure.
Pod x Plane Heatmap: Rows are pods, columns are planes. A bright cell indicates a pod-local skew — one pod's traffic lands disproportionately on one plane. Check placement and routing for that pod.
Rail x Plane Heatmap: Rows are rails (railIndex), columns are planes. A bright column on one rail indicates a rail-specific skew. Common causes include asymmetric ECMP hash or a misconfigured rail link.
Role x Plane Heatmap: Rows are device roles (leaf, spine, superspine), columns are planes. A bright cell on a specific role narrows whether the skew is spine-side or leaf-side.
Interface Detail
Top Skewed Interfaces: A ranked table of interface endpoints by TX+RX traffic (trafficGB, ifTcTxOctet, ifPgRxOctet), including plane, deviceName, ifName, and egressDeviceName. Use it to identify the specific links contributing the most to the observed imbalance.
Multi-Plane Overview – ECN RoCE Congestion
Use this dashboard when traffic is moving but ECN marking or pause backpressure is elevated. ECN and pause are congestion-feedback signals, not packet-loss signals — elevated values here do not imply drops. Navigate to Packet Drop Root Cause if drops are also high.
ECN Marking
ECN by Plane and TC: A heatmap with planes on one axis and traffic classes on the other, using ifTcTxEcnMarkedTc. Use it to determine whether marking is class-local (only one TC is marked) or plane-local (all TCs on one plane are marked).
ECN by Plane: A bar chart of switch-side ECN marks (ifTcTxEcnMarkedTc) summed per plane over time. Use it to track whether ECN pressure is rising or falling after a configuration change.
Pause Backpressure
Pause by Plane: A bar chart of pauseFramesTx + pauseFramesRx by plane. Pause frames are PFC link-level backpressure and indicate lossless congestion feedback is active. A plane with consistently high pauses relative to others may have a slower receiver or a misconfigured headroom buffer.
Interface Detail
Top ECN Interfaces: A ranked table of interfaces by ifTcTxEcnMarkedTc with drops included as context. Columns include plane, deviceName, ifName, egressDeviceName, and trafficClass. Use it to localize which switch ports are generating the most marking.
Multi-Plane Overview – Packet Drop Root Cause
Use this dashboard when drops are the dominant signal. It classifies the drop type (WRED, HoQ, link-down, PG buffer, shared buffer, TX no-buffer, interface discards), locates loss by plane and role, and ranks the specific interfaces and pod/rail/role scopes responsible.
Drop Classification
Drops by Reason: A table comparing drop counter families across planes over time. Columns include totalDrops, redDropPackets (WRED), egressHoQDiscards (head-of-queue), egressLinkDownDiscards, pgRxBufferDiscard, pgRxSharedBufferDiscard, txNoBufferDiscardUc, ifInDiscards, and ifOutDiscards. Identifying which counter type is elevated narrows the root cause before looking at individual devices.
Drop Localization
Plane / Role Drop Heatmap: A heatmap with planes on one axis and device roles on the other, using totalDrops. A bright cell on leaf/P2 indicates leaf-side drops on that plane; a bright cell on spine/P1 indicates spine-side loss.
Interface Detail
Top Dropping Interfaces: A ranked table of interface endpoints by totalDrops with redDropPackets, egressHoQDiscards, egressLinkDownDiscards, ifInDiscards, and ifOutDiscards as supporting columns. Includes plane, deviceName, ifName, egressDeviceName, and role. Use it to identify the exact links responsible for the observed loss.
Pod / Rail / Role Focus: A ranked table of plane, groupPod, railIndex, and role combinations by totalDrops. Use it when the top-dropping interfaces span many devices and the investigation needs to stay fabric-oriented rather than device-oriented.
Multi-Plane Overview – Queue Analytics
Use this dashboard when packets are delayed or buffered rather than dropped. Queue wait and buffer occupancy are the signals here; packet-drop counters are absent by design. Navigate to Packet Drop Root Cause if you need to classify actual packet loss.
Egress Wait
txWait by Plane: A bar chart of txWait by plane over time. Egress wait is the first queueing symptom before drops are necessary to prove loss; a rising txWait on one plane while others are flat indicates a plane-local egress bottleneck.
Buffer Occupancy
Ingress Buffer Histogram: A heatmap of ingress buffer-depth distribution (histogramInterfaceIngressBuffer from histogram_v1), grouped by plane. The histogram shows which occupancy buckets receive traffic — a shift toward higher buckets indicates packets dwelling longer at ingress.
Egress Buffer Histogram: A heatmap of egress queue and buffer-depth distribution (histogramInterfaceEgressBuffer from histogram_v1), grouped by plane. Sustained high egress occupancy often precedes WRED or HoQ drops.
PG Pressure
Priority-Group Pressure: A heatmap of priorityGroupPressure by plane and priority group, derived from receive octets plus PG buffer discards. Use it when a specific priority group (especially a lossless PG) is suspected of causing congestion rather than the whole plane.
Interface Detail
Top Congested Interfaces: A ranked table of interfaces by txWait, including plane, deviceName, and ifName. Use it to identify the exact switch ports accumulating egress wait.
Multi-Plane Overview – NIC RoCE Metrics
Use this dashboard to correlate switch-level plane symptoms with server NIC congestion-control behavior. All data comes from system_v1 Amber component telemetry filtered to eventSource=NIC and type=component.
The heatmap uses a log-scaled per-sample CC symptom heat value (ccSymptomHeat = log10(1 + ccSymptomIntensity)) so that a high-volume plane 0 does not visually obscure planes 1–3.
NIC CC Symptoms
NIC CC Symptoms Heatmap: A plane-first heatmap with planes on one axis and server host names (deviceName) on the other, colored by ccSymptomHeat. Supporting measures available for drilldown include ccSymptomIntensity, ccSymptoms, cnpFeedback, and the raw Amber counters hwNpCnpSent, hwRpCnpHandled, hwRoceAdpRetrans, hwRoceAdpRetransTo, hwOutOfSequence, and localAckTimeoutErr. Use this panel when ECN or pause is high on a specific plane to confirm whether NIC-side CC symptoms correspond to the same plane or to specific servers.
Last updated: