NVIDIA Minerva flags an anomaly when incoming telemetry deviates significantly from a learned baseline. Baselines are built continuously from time-series data and log streams.
Log-Based Anomalies
Analyzes text streams (for example, syslogs) for issues that metrics alone might miss.
|
Type |
How it works |
|---|---|
|
Zero Day Log Anomalies (Unsupervised) |
NLP identifies rare or never-before-seen log patterns, useful for novel bugs or unexpected hardware failures |
|
Collective Learning (Supervised) |
Flags high-impact events based on known error signatures, for example, GPU XID errors, PCIe AER errors |
Metric-Based Anomalies
Analyzes numerical data for performance or hardware health deviations.
|
Type |
How it works |
|---|---|
|
Individual Anomalies |
Detects spikes or dips relative to a component's own historical baseline |
|
Group Anomalies |
Compares one component (for example, a single GPU) against its peers to identify behavioral outliers |
Examples
Network and Fabric Health
-
Pause Frames RX (
pauseFramesRX): detects abnormal surge in pause frames on NIC-facing switch interfaces. Use this to identify a straggler node that is slowing fabric throughput for the whole cluster.
-
ROCE Adaptive Retransmits (
hwRoceAdpRetrans): detects abnormal retry rates from a host's HCA. Helps distinguish transient noise from sustained congestion.
Hardware and System Health
-
GPU Temperature (Peer Comparison): identifies GPUs with a thermal profile substantially different from peers running the same workload. Locates nodes with cooling issues or localized hardware defects.
-
Syslog Rates (Log Storms): detects sudden surges in log volume generated by a host. Detects software loops or misconfigured applications flooding system resources.
Anomaly Terminology
|
Term |
Definition |
|---|---|
|
Base Type |
The most granular entity being modeled, for example, |
|
Instance Key |
Globally unique identifier for a specific component, for example, |
|
Aggregate Object |
Defines how data is grouped and which algorithm is applied, for example, grouping all GPU temperatures across a pod to find outliers |
|
Confidence Score |
Probability (0–1) that a deviation is a true anomaly |
Troubleshooting and Remediation
NVIDIA Minerva provides "Rare Log" anomalies that generate remediation guidance, including step-by-step workflows to diagnose and fix issues. You can adjust these configurations by editing the anomaly-detection.json file.
Last updated: