Export Metrics to External Prometheus | NVIDIA Minerva User Guide

Export NVIDIA Minerva metrics to an external Prometheus via remote-write so the Professor-X Grafana dashboards can query them. This is opt-in, off by default, and runs alongside the normal ingest path without affecting it.

Overview

A commented, ready-to-edit configuration file ships in the gateway-push setup.

In the repo, the seed file is mgmt/etc/otel-collector/gateway-push.yaml.
On a deployed node, mgmt stages it as the live copy at /opt/minerva/etc/otel-collector/gateway-push.yaml.
Copy the live configuration into the validation sandbox and make any edits.
Validate the changes.
Copy the sandbox configuration back to the live copy.

processors:
  # Export only the families the dashboards use. Remove to export everything.
  filter/professorx_scope:
    error_mode: ignore
    metrics:
      metric:
        - >
          not (
            IsMatch(name, "^nvswitch_.*") or
            IsMatch(name, "^nvidia_smi_.*") or
            IsMatch(name, "^DCGM_.*") or
            IsMatch(name, "^hw_.*") or
            IsMatch(name, "^port_(rx|tx)_bytes$") or
            IsMatch(name, "^Port.*") or
            IsMatch(name, "^slurm_.*") or
            IsMatch(name, "^MAX_RTT_COUNTER.*") or
            IsMatch(name, "^tx_prio.*") or
            IsMatch(name, "^(AI_INC|HAI_INC|HYPER_DEC|TX_DEC|DEC)_COUNTER.*")
          )
  batch/professorx:
    send_batch_size: 8192
    send_batch_max_size: 8192
    timeout: 5s

exporters:
  prometheusremotewrite/professorx:
    endpoint: http://<prometheus-host>:9090/api/v1/write   # EDIT THIS
    timeout: 30s
    tls:
      insecure: true            # dev only — use TLS + auth in production
    add_metric_suffixes: false  # keep the original metric names
    resource_to_telemetry_conversion:
      enabled: true             # promote resource attributes to labels
    target_info:
      enabled: false

service:
  pipelines:
    metrics/professorx:
      receivers: [otlp, prometheusremotewrite]
      processors: [memory_limiter, transform/gauge_to_sum, filter/professorx_scope, batch/professorx]
      exporters: [prometheusremotewrite/professorx]

Enable

All commands run on the deployed node.

minerva-cli otelcol --config <name> resolves <name> from /opt/minerva/var/utility/, so edit and validate a copy there first, then place the validated file into the live configuration.

Start the target Prometheus with --web.enable-remote-write-receiver.
Copy the live configuration into the validation sandbox and edit it there. Set endpoint and uncomment the block.

cp /opt/minerva/etc/otel-collector/gateway-push.yaml \
   /opt/minerva/var/utility/gateway-push.yaml
$EDITOR /opt/minerva/var/utility/gateway-push.yaml

Validate it, and optionally preview the merged configuration, before applying.

minerva-cli otelcol validate --personality gateway-push --config gateway-push.yaml
NO_TTY=1 minerva-cli otelcol show --personality gateway-push --config gateway-push.yaml | grep -A20 'metrics/professorx'

Once validation passes, copy it to the live configuration. mgmt publishes it to Consul, and the collector reloads within seconds.

cp /opt/minerva/var/utility/gateway-push.yaml \
   /opt/minerva/etc/otel-collector/gateway-push.yaml

Verify

P=http://<prometheus-host>:9090
curl -s "$P/api/v1/label/__name__/values" | jq '.data | length'
curl -s "$P/api/v1/label/__name__/values" | jq '[.data[] | select(startswith("nvswitch_"))] | length'

Rollback

Re-comment the block in the setup configuration. mgmt republishes it, and the metrics/professorx pipeline goes away.

Notes

The filter/professorx_scope allowlist controls volume. Without it, the full metric firehose is pushed.
The allowlist keeps two distinct port families:
- Lowercase counters: ^port_(rx|tx)_bytes$
- Uppercase counters: ^Port.*

VictoriaMetrics Target

VictoriaMetrics includes a native OTLP/HTTP receiver. Configure the otlphttp/professorx exporter to send metrics directly to VictoriaMetrics by setting metrics_endpoint to the VictoriaMetrics OTLP metrics path.

Important: Use the OTLP path, not the Prometheus remote-write path. For example, set metrics_endpoint to http://<victoriametrics-host>:<port>/opentelemetry/v1/metrics.

The configuration below streams metrics to VictoriaMetrics and normalizes OTLP attribute names to the underscore-based label names used by the Professor-X Grafana dashboards.
- transform/professorx_labels is required whenever the Professor-X dashboards query VictoriaMetrics.
- VictoriaMetrics stores OTLP attribute names verbatim. Without this transform, attributes such as net.host.name remain dotted instead of becoming net_host_name.
- Most dashboard panels use net_host_name as the primary host or switch identity. If the label is not rewritten, those panels return No data.

The complete VictoriaMetrics configuration:

processors:
  # REQUIRED for the Professor-X dashboards on VM: rewrite OTLP dotted attribute
  # keys (net.host.name, port.type, …) to Prometheus-style underscores
  # (net_host_name, port_type). Without it VM keeps the dotted names and the
  # dashboards' host/switch panels render nothing. Doing the rename here (rather
  # than via VM's -opentelemetry.usePrometheusNaming) keeps counter names bare.
  transform/professorx_labels:
    error_mode: ignore
    metric_statements:
      - context: resource
        statements: [ replace_all_patterns(attributes, "key", "\\.", "_") ]
      - context: datapoint
        statements: [ replace_all_patterns(attributes, "key", "\\.", "_") ]
  # Export only the families the dashboards use (same allowlist as the
  # remote-write example above). Remove to export everything.
  filter/professorx_scope:
    error_mode: ignore
    metrics:
      metric:
        - >
          not (
            IsMatch(name, "^nvswitch_.*") or IsMatch(name, "^nvidia_smi_.*") or
            IsMatch(name, "^DCGM_.*") or IsMatch(name, "^hw_.*") or
            IsMatch(name, "^port_(rx|tx)_bytes$") or IsMatch(name, "^Port.*") or
            IsMatch(name, "^slurm_.*") or IsMatch(name, "^MAX_RTT_COUNTER.*") or
            IsMatch(name, "^tx_prio.*") or
            IsMatch(name, "^(AI_INC|HAI_INC|HYPER_DEC|TX_DEC|DEC)_COUNTER.*")
          )
  batch/professorx:
    send_batch_size: 8192
    send_batch_max_size: 8192
    timeout: 5s
exporters:
  otlphttp/professorx:
    metrics_endpoint: "http://<victoriametrics-host>:<port>/opentelemetry/v1/metrics"  # EDIT THIS
    timeout: 30s
    tls:
      insecure: true            # dev only — use TLS + auth in production
service:
  pipelines:
    metrics/professorx:
      receivers: [otlp, prometheusremotewrite]
      # transform/professorx_labels must sit before batch (batch is last); see
      # "Processor ordering" below.
      processors: [memory_limiter, transform/gauge_to_sum, filter/professorx_scope, transform/professorx_labels, batch/professorx]
      exporters: [otlphttp/professorx]

Last updated: October 15, 2026