The following sections guide Infra Ops, SREs, and Application owners through the Operational Readiness Review (ORR) of an AI Factory deploying with NVIDIA Minerva. An ORR confirms that the NVIDIA Minerva stack is ingesting telemetry from every configured data source and is ready to observe the AI Factory cluster it monitors. The ORR begins after the NVIDIA Minerva platform and collector stacks are installed (see Deployment and Installation) and AI Factory data sources are onboarded, and ends with an initial deployment sign-off indicating that the cluster is ready for workloads.
Outcomes of a Successful ORR
By the end of the ORR, the operator team produces:
-
A Pass Criteria checklist for each of the ten validation areas below.
-
An archived idle baseline: screenshots of the AI Cluster Health, GPU Summary by Node, NVLink Health Dashboard, Professor-X Dashboard, and Telemetry Health dashboards captured while the cluster is idle. These are the reference points every later "is this normal?" question will be measured against.
-
A gap list: anything not green at sign-off, which carries forward to the next operations phase.
Audience
|
Persona |
Day 0 focus |
|---|---|
|
Infrastructure / DC Ops |
Physical fabric, optics, cabling, power and thermal sensors. |
|
System Administrators / SREs |
Node, NIC, GPU, NVSwitch, kernel-log, and telemetry-pipeline sanity. |
|
Network Engineers |
Leaf/Spine/Super Spine reachability, link state, congestion control, ECMP balance. |
|
AI/ML Engineers / Application Owners |
Job ingestion, scheduler integration, idle GPU baselines for later workload comparison. |
|
Tenant Admins |
Tenant-level inventory totals and per-tenant grouping ( |
Prerequisites
The following must be true before you begin the ORR:
-
NVIDIA Minerva Platform Stack is installed and reachable (see Platform Installation).
-
NVIDIA Minerva Collector Stack is installed on-premises and the gRPC tunnel is established (see Collector Installation).
-
AI Factory data sources are onboarded (see Connect to Data Sources): DOCA Telemetry on hosts, Cumulus OTLP/syslog, NMX-T, Slurm event collectors, and Slurm logs.
-
The operator has UI access to both
default(admin) anduser-tenanttenants and knows which tenant each dashboard set imports into. -
The Reference Configuration Package is available on a workstation that can reach the NVIDIA Minerva UI in the browser (required for the dashboard import in Pre-Validation Setup).
If any prerequisite is not met, stop and complete it first: ORR validation against an incomplete platform produces false-negative or false-positive results.
Section Structure
Each subsection that follows uses the same structure already used by Preconfigured Dashboards:
-
Persona Use Cases: who cares about this validation step and why.
-
Validation Procedure: numbered, imperative steps the operator follows in the UI.
-
Reference Dashboards and Sections: cross-references to the preconfigured dashboards and alerts.
-
Pass Criteria: checklist to complete before advancing to the next area.
-
Common Pitfalls: known failure modes and how they appear in the dashboards.
Validation Areas
The recommended order runs from the foundation up: confirm topology and telemetry first, then physical and logical layers, then workload validation. The subsections can be performed in any order, but later sections often depend on earlier ones being green.
|
Validation area |
Primary dashboards |
|---|---|
|
n/a — setup |
|
|
Trends, Log Insights (no dedicated preconfigured dashboard yet) |
|
|
Performance Dashboard (imported from the Reference Configuration Package), Professor-X Dashboard |
|
|
Aggregate sign-off |
What This Guide Does Not Validate
These are explicitly out of scope for ORR:
-
Anomaly configurations. A subset of potential anomalies are shipped enabled out-of-the-box. Deciding which detectors to turn on, and tuning their thresholds, is a post-ORR activity.
-
Notifications endpoint configurations.
-
Auto Correlation namespace and view binding.
-
Production workload baselines. ORR captures idle baselines only. Active-workload baselines are captured once the cluster is running representative jobs.
Last updated: