Verifies that the AI Factory cluster topology has been correctly discovered and modeled: every switch, NIC, NVSwitch, and compute node is present and grouped into the expected Pods and Scalable Units (SUs). Topology validation is the foundation ORR validation step — every other subsection's results depend on the underlying inventory being right.
Persona Use Cases
|
Persona |
Primary Use Case |
|---|---|
|
Infrastructure / DC Ops |
Confirms every physical asset on the build sheet is reporting telemetry. |
|
Network Engineers |
Confirms leaf, spine, and super-spine switch counts and their interconnect link counts match design. |
|
Cluster Administrators |
Confirms node counts and SU/Pod groupings match scheduler configuration. |
|
Tenant Admins |
Verifies tenant-scoped inventory totals ( |
Validation Procedure
Step 1: Compare Device Counts to the Build Sheet
-
In the
defaulttenant, open the Telemetry Health dashboard. -
Read the Device Counts summary:
Total Device Count,Switches,Nodes,Node DCGM,OpenTelemetry Metrics,SLURM Records,SLURM Node Event,Switch Histograms,Link State Sources. -
Compare each value against the deployment plan. Any zero or significantly low value is a topology gap.
Step 2: Verify Interface Counts per Fabric Tier
-
Scroll to Interface Counts.
-
Confirm the per-tier interface counts match the wiring plan:
-
Switch-to-Switch(Leaf-to-Spine, Spine-to-Leaf, Spine-to-Super Spine, Super Spine-to-Spine) -
Switch-to-Node(Leaf-to-NIC, NIC-to-Leaf)
-
-
Per-tier mismatches usually indicate that LLDP or BGP-LS adjacency has not yet been fully learned for some devices.
Step 3: Validate Pods and SUs
-
In the
user-tenant, open the Professor-X Dashboard. -
Locate the SU Activity widget under Performance & Fabric Bandwidth and the Live Switches & Nodes counters under Global Inventory & Telemetry Health.
-
Confirm every expected SU has at least one entry, and that switch and node live counts equal the discovered totals from Step 1.
Step 4: Spot-check Tenant Grouping Enrichment
-
Pick one node and one switch on any panel that exposes the
groupCustomorgroupNamesdimensions (the Trends page allows adding either to a view). -
Confirm both fields are populated. Empty values indicate that the topology parent-walk has not yet completed for that device or that the device is not associated with a Group entity.
Reference Dashboards and Sections
-
Telemetry Health: Device Counts, Interface Counts.
-
Professor-X Dashboard: SU Activity, Global Inventory & Telemetry Health.
-
AI Cluster Health: used for cross-checking when device counts look wrong.
-
Configuration sources:
clusterName,deviceName,eventSourcemappings (NVIDIA Minerva Ingestion → Dimensions & Measures Mapping).
Pass Criteria
-
[ ]
Total Device Countequals the sum of expected switches and nodes from the build sheet. -
[ ]
Switchescount matches the planned leaf, spine, super-spine, and NVSwitch totals. -
[ ]
Nodescount matches the planned compute-node total. -
[ ]
Node DCGMcount equals the planned GPU-bearing node count. -
[ ]
SLURM RecordsandSLURM Node Eventcounters are non-zero. -
[ ] All four
Switch-to-Switchinterface counts (Leaf-to-Spine,Spine-to-Leaf,Spine-to-Super Spine,Super Spine-to-Spine) match the wiring plan. -
[ ]
Leaf-to-NICandNIC-to-Leafcounts match the wiring plan. -
[ ] Every SU appears in the Professor-X SU Activity widget.
-
[ ] Spot-checked nodes and switches have non-empty
groupCustomandgroupNamesvalues. -
[ ] Idle-baseline screenshots of Telemetry Health and Professor-X Dashboard Inventory archived.
Common Pitfalls
-
Switch counts low while node counts correct (or vice versa). Usually traces to a single OID configuration file not yet loaded, or a Consul KV path not yet populated. Switch and node ingestion paths are independent.
-
Interface counts at zero with non-zero device counts. LLDP or BGP-LS has not yet been learned. Interface entities depend on a parent Device entity already existing: NVIDIA Minerva will defer interface ingestion if the device hasn't been written, and replay them when the device shows up.
-
Devices in
MAINTENANCEstate appear missing. The Meta-cache layer can mark devices asMAINTENANCE(for example, from NOS API) without removing them from the topology. If a device is unexpectedly not on a panel, check whetherdeviceAdminState = MAINTENANCEis filtering it out. -
groupCustomis populated butgroupNamesis empty. The parent-walk has run for the immediate parent but not all the way up to the Group entity. Re-check that the device is actually attached to a Group in the topology model.
Last updated: