GPU Health | NVIDIA Minerva User Guide

Verifies that DCGM-sourced GPU telemetry is flowing for every expected GPU, that idle thermal and power readings are within vendor specification, and that no XID errors are firing on an otherwise-quiet cluster. This is the canonical "are my GPUs ready for work" check.

Persona Use Cases

Persona	Primary Use Case
DC Operations	Identifies racks or host enclosures exceeding power or thermal thresholds at idle.
Reliability Engineer (SRE)	Investigates any non-zero XID counts or thermal-throttling events to determine if a GPU needs RMA before workloads land on it.
Hardware Engineering	Validates cooling efficiency and power distribution across the rack using the idle baseline.
AI/ML Engineers	Confirms no straggler GPUs are present that would imbalance future training jobs.

Validation Procedure

Step 1: Confirm DCGM Ingestion for Every Node

In the user-tenant, open the GPU Summary by Node dashboard.
Confirm every expected node appears in the per-node tables and trends. A node missing from this dashboard but present in Telemetry Health indicates DCGM is not exporting on that node.
Cross-check against Node DCGM from Telemetry Health > Device Counts (Telemetry Health > Device Sources). The two should agree.

Step 2: Validate Idle Thermal and Power Baselines

On GPU Summary by Node > Temperature and Power, inspect:
- GPU Max Temperature by Node+GPU heatmap: every cell should be below the GPU Temperature High threshold of 75°C used by the Preconfigured Alerts.
- GPU Max Power by Node+GPU: no node should be drawing power consistent with active workload.
On GPU Summary by GPU > Temperature and Power, inspect:
- GPU Power LC by GPU: instantaneous power per GPU. No GPU should be consistently hitting power limits.
- GPU Temperature LC by GPU: uniform low across all GPUs.

Step 3: Verify XID Error Stream Is Empty

On GPU Summary by GPU > Errors, inspect GPU NonZero XID by GPU. The heatmap should be empty across the validation window.
On GPU Summary by Node > Errors, inspect Nodes with non-zero XID errors. The chart should be empty.
Open AI Cluster Health > Health Issues (Summary Tiles). Confirm GPU Critical Errors reads 0.

Step 4: Cross-check GPU Utilization at Idle

On GPU Summary by GPU > Utilization and GPU Summary by Node > Utilization, confirm utilization and memory utilization are uniformly near zero.
Any GPU showing sustained utilization at idle is either running an unscheduled process or has stuck driver state — both worth investigating before workloads run.

Step 5: Capture the Baseline

Archive screenshots of GPU Summary by Node Utilization, Temperature and Power, and Errors panels for the agreed idle window.
These become the reference for distinguishing "this is busy" from "this is wrong" once jobs start.

Reference Dashboards and Sections

GPU Summary by GPU: Utilization, Temperature and Power, Errors, Metric Reference Mapping.
GPU Summary by Node: Utilization, Temperature and Power, Errors, Metric Reference Mapping.
AI Cluster Health: Health Issues GPU Critical Errors, Key Metrics GPU Temperature High.
Preconfigured Alert GPU Temp High (Preconfigured Alerts): same 75°C threshold.

Source Metric Mapping (DCGM → NVIDIA Minerva)

Dashboard label	Native NVIDIA metric
Utilization	`DCGM_FI_DEV_GPU_UTIL`
Memory Used	`DCGM_FI_DEV_FB_USED`
Temperature	`DCGM_FI_DEV_GPU_TEMP`
Power Usage	`DCGM_FI_DEV_POWER_USAGE`
XID Errors	`DCGM_FI_DEV_XID_ERROR`

Pass Criteria

[ ] Every expected node and GPU appears on GPU Summary by Node and GPU Summary by GPU.
[ ] No GPU exceeds 75°C during the idle window.
[ ] No GPU is drawing power inconsistent with idle.
[ ] GPU NonZero XID by GPU heatmap is empty for the idle window.
[ ] GPU Critical Errors summary tile on AI Cluster Health reads 0.
[ ] Utilization is uniformly near zero across all GPUs.
[ ] Idle-baseline screenshots archived.

Common Pitfalls

One node missing GPUs in the dashboard. DCGM exporter is not running, or the OTel collector on that node is not subscribed to the DCGM endpoint. Validate by connecting via SSH to the node and confirming dcgm-exporter is running and reachable.
All GPUs on one node showing the same XID burst. Often a fabric-manager or driver state event affecting the whole HGX board, not a per-GPU hardware fault.
Persistent low-level utilization. Stuck CUDA contexts from prior testing, or nvidia-smi-style probes from monitoring agents. Distinguish by checking for active processes on the host.
Temperature appears uniform across nodes but is too high. This is the harder signal to catch during the ORR: the cluster is technically passing the per-GPU 75°C threshold but is room-air or cooling-circuit constrained. Document the absolute idle temperature so DC Ops can compare against vendor environmental specs.

Last updated: October 14, 2026