Ingestion by Source
This section categorizes the origin of incoming telemetry to help isolate chatty or silent components.
-
Ingestion by Source (Metric Count): A breakdown of total points by
EVENTSOURCE.-
Switch: Telemetry from the Ethernet leaf/spine fabric.
-
NVLink: Telemetry from the NVSwitch/NVLink fabric.
-
NIC: Compute host interface telemetry (InfiniBand/Ethernet).
-
DCGM: GPU-specific metrics including power, temperature, and XID events.
-
TSDB Datasources (Database Counts)
This section tracks the health of time-indexed metric and event records across the platform's datasources. These are stored in a Druid-based TSDB layer with specific retention and routing logic.
|
Datasource |
Role and Content |
Typical Retention |
|---|---|---|
|
|
Primary store for general metrics and events (SNMP, JSON, event streams). |
~120 days |
|
|
Packet-level drop events from WJH (What Just Happened) telemetry. |
~7 days |
|
|
Syslog-derived time series and syslog-tagged telemetry. |
~30 days |
|
|
System objects: components, queues, optical lanes, probes, and links. |
~10 days |
|
|
Interface-scoped metrics (network interfaces as first-class objects). |
~10 days |
Ingestion Health and Limits
-
Ingestion Rates (Last 6H vs. 7D): Compares short-term spikes for
system_v1,syslog_v1, andraw_v1against historical baselines. -
MinervaHealth-ingestionRateExceeded: A critical monitor for Kafka topics where the ingestion rate exceeds configured platform limits (defined in
mgmt/etc/global.json). Sustained alerts indicate a need to scale ingestion workers or adjust limits. -
Otel-ingestion-rate: Measures the throughput of OpenTelemetry (OTLP) data, which carries the bulk of GPU and node telemetry.
Network and Receiver Transport
-
LB-UDP-Drops: Displays packets dropped at the load balancer level. High drops indicate the NVIDIA Minerva ingestion engine is overwhelmed and cannot keep up with the UDP stream from the fabric.
Dimensions and Measures Mapping
The following mappings translate AI factory telemetry into NVIDIA Minerva-native dimensions:
|
NVIDIA Dimension |
NVIDIA Minerva Source |
Description |
|---|---|---|
|
|
|
Identifies the specific AI factory or cluster instance. |
|
|
|
The hostname or switch ID of the data originator. |
|
|
|
Categorizes data into |
Last updated: