Validate Topology and Inventory (Pods, SUs)

Verifies that the AI Factory cluster topology has been correctly discovered and modeled: every switch, NIC, NVSwitch, and compute node is present and grouped into the expected Pods and Scalable Units (SUs). Topology validation is the foundation ORR validation step — every other subsection's results depend on the underlying inventory being right.

This validation area is part of the AI Factory Operational Readiness Review. Before you begin, complete the steps in Prepare for Readiness Validation.

Persona Use Cases

Persona	Primary Use Case
Infrastructure / DC Ops	Confirms every physical asset on the build sheet is reporting telemetry.
Network Engineers	Confirms leaf, spine, and super-spine switch counts and their interconnect link counts match design.
Cluster Administrators	Confirms node counts and SU/Pod groupings match scheduler configuration.
Tenant Admins	Verifies tenant-scoped inventory totals (`groupCustom`, `groupNames`) for chargeback and access boundaries.

Validation Procedure

Step 1: Compare Device Counts to the Build Sheet

In the default tenant, open the Telemetry Health dashboard.
Read the Device Counts summary: Total Device Count, Switches, Nodes, Node DCGM, OpenTelemetry Metrics, SLURM Records, SLURM Node Event, Switch Histograms, Link State Sources.
Compare each value against the deployment plan. Any zero or significantly low value is a topology gap.

Step 2: Verify Interface Counts per Fabric Tier

Scroll to Interface Counts.
Confirm the per-tier interface counts match the wiring plan:
- Switch-to-Switch (Leaf-to-Spine, Spine-to-Leaf, Spine-to-Super Spine, Super Spine-to-Spine)
- Switch-to-Node (Leaf-to-NIC, NIC-to-Leaf)
Per-tier mismatches usually indicate that LLDP or BGP-LS adjacency has not yet been fully learned for some devices.

Step 3: Validate Pods and SUs

In the user-tenant, open the Professor-X Dashboard.
Locate the SU Activity widget under Performance and Fabric Bandwidth and the Live Switches & Nodes counters under Global Inventory and Telemetry Health.
Confirm every expected SU has at least one entry, and that switch and node live counts equal the discovered totals from Step 1.

Step 4: Spot-check Tenant Grouping Enrichment

Pick one node and one switch on any panel that exposes the groupCustom or groupNames dimensions (the Trends page allows adding either to a view).
Confirm both fields are populated. Empty values indicate that the topology parent-walk has not yet completed for that device or that the device is not associated with a Group entity.

Reference Dashboards and Sections

Telemetry Health: Device Counts, Interface Counts.
Professor-X Dashboard: SU Activity, Global Inventory and Telemetry Health.
AI Cluster Health: used for cross-checking when device counts look wrong.
Configuration sources: clusterName, deviceName, eventSource mappings (NVIDIA Minerva Ingestion → Dimensions & Measures Mapping).

Pass Criteria

[ ] Total Device Count equals the sum of expected switches and nodes from the build sheet.
[ ] Switches count matches the planned leaf, spine, super-spine, and NVSwitch totals.
[ ] Nodes count matches the planned compute-node total.
[ ] Node DCGM count equals the planned GPU-bearing node count.
[ ] SLURM Records and SLURM Node Event counters are non-zero.
[ ] All four Switch-to-Switch interface counts (Leaf-to-Spine, Spine-to-Leaf, Spine-to-Super Spine, Super Spine-to-Spine) match the wiring plan.
[ ] Leaf-to-NIC and NIC-to-Leaf counts match the wiring plan.
[ ] Every SU appears in the Professor-X SU Activity widget.
[ ] Spot-checked nodes and switches have non-empty groupCustom and groupNames values.
[ ] Idle-baseline screenshots of Telemetry Health and Professor-X Dashboard Inventory archived.

Common Pitfalls

Switch counts low while node counts correct (or vice versa). Usually traces to a single OID configuration file not yet loaded, or a Consul KV path not yet populated. Switch and node ingestion paths are independent.
Interface counts at zero with non-zero device counts. LLDP or BGP-LS has not yet been learned. Interface entities depend on a parent Device entity already existing: NVIDIA Minerva will defer interface ingestion if the device hasn't been written, and replay them when the device shows up.
Devices in MAINTENANCE state appear missing. The Meta-cache layer can mark devices as MAINTENANCE (for example, from NOS API) without removing them from the topology. If a device is unexpectedly not on a panel, check whether deviceAdminState = MAINTENANCE is filtering it out.
groupCustom is populated but groupNames is empty. The parent-walk has run for the immediate parent but not all the way up to the Group entity. Re-check that the device is actually attached to a Group in the topology model.

Last updated: October 30, 2026