Networking Minerva User Guide

Install on a Single Node

This guide covers deploying the NVIDIA Minerva platform on a single-node Docker Swarm. Services are managed by the svm script and grouped into stacks. For the stacks that make up the platform, see Docker Swarm - Platform.

Before you begin, make sure the host meets the Resource Requirements and is prepared as described in Prepare Hosts for Docker Swarm.

The installer can run in three deployment types:

Deployment Type

Description

Unified

Platform and collector services are deployed in the same installation.

Platform

Platform Stack services for UI, API, backend, infrastructure, and storage.

Collector

On-premises Collector Stack services, responsible for data collection and forwarding data to the Platform Stack.

Install in this order:

  1. Run the installer on the manager node.

  2. Bring up the stacks.

Run the Installer on the Manager Node

  1. Copy the installer package to the manager node and run it:

    chmod +x installer.minerva.<BUILD>
    ./installer.minerva.<BUILD>
    

NVIDIA Minerva has three installer versions. If you aren't using the standard installer, use its file name in the command above:

Installer Version

Description

installer.minerva.<BUILD>

Standard installer. Doesn't include the Docker images.

installer.minerva.all.<BUILD>

Full installer. Includes all required Docker images.

installer.minerva.collector.<BUILD>

Collector installer. Includes all required NVIDIA Minerva collector Docker images.

On ARM64 hosts, use the ARM64 builds of the bundled installers, which have arm64 in the name: installer.minerva.all.arm64.<BUILD> and installer.minerva.collector.arm64.<BUILD>. An installer run on a host of the other architecture stops, and its message names the installer to use.

  1. Answer the installer prompts. Enter ? at any prompt for a description of that parameter. Answers are saved, so re-runs (upgrades) skip previously answered prompts. For what each prompt means, see Installer Prompts.

  2. When the installer finishes, go to the installation directory. The software is installed under ~/minerva/:

    cd ~/minerva
    
  3. For a hybrid installation, create a bootstrap bundle for each collector (see Create a Collector Bootstrap Bundle in Install Collectors for a Hybrid Deployment).

Installer Prompts

The prompts are listed in the order the installer asks them. Some appear only for certain deployment types, or only when a value isn't already set.

Deployment type

Select deployment type (? for help)
 1. platform
 2. collector
 3. unified
Choose an option:

Refer to Deployment Models for details on deployment types. For a hybrid installation, the Platform Stack must be fully installed before beginning the Collector Stack installation.

The collector installer (installer.minerva.collector.<BUILD>) doesn't ask this question. It always installs a collector deployment.

Container registry base path

Container registry base path (? for help) :

The Docker registry path the installer pulls images from, for example registry.example.com/<path>/. Do not include http:// or https://. If the registry requires authentication, run docker login on the node before you run the installer.

  • NMC: Use nvcr.io.

  • With a bundled installer (installer.minerva.all.<BUILD> or installer.minerva.collector.<BUILD>), or when you pass -p <registry-base>, the installer doesn't ask for a registry.

Platform Stack IP address or FQDN

Enter Platform Stack IP address or FQDN (? for help) [<host-name>]:

Platform deployments only. This is the address collectors use to reach the Platform Stack, and the platform's TLS certificate is generated for it. The default is the host name that hostname -f returns on this host. Enter an IP address or FQDN that every collector node can resolve and reach.

If you enter a host name, it must resolve through DNS. An entry in a collector's /etc/hosts isn't enough, because the collector's services run in containers that don't read it.

Kafka external IP address or FQDN

Kafka external FQDN or IP address for TLS listener (? for help) [<platform-address>]:

Platform deployments only. This is the address collectors use to reach Kafka on port 9093. The default is the Platform Stack address. Change it only if Kafka runs on a different node.

Additional platform TLS SANs

Additional platform TLS SANs comma-separated. Hit Enter if none (? for help) :

localhost, 127.0.0.1, and the platform and Kafka addresses are always included. Enter each SAN as DNS:<name> or IP:<address>, or as a bare name or address.

WJH gNMI dial-out username

WJH gNMI dial-out username. Hit Enter for the read-only default (? for help) [telemetry-ro]:

WJH gNMI dial-out password

WJH gNMI dial-out password. Hit Enter to leave unconfigured (? for help) :

Time-series storage type

Where would you like to save the time series data? (? for help)
 1. local
 2. aws-s3
 3. s3-compatible
 4. azure
Choose an option [local]:

Where NVIDIA Minerva stores time-series data.

  • local: on the host. There are no further prompts.

  • aws-s3: Amazon S3. The installer asks for Time series S3 path (an s3:// URI) and, if the AWS region isn't already set, Runtime AWS region for S3 access.

  • s3-compatible: an S3-compatible object store that you provide, for example Ceph. The installer asks for Customer S3-compatible endpoint (host:port), Customer S3-compatible endpoint protocol (http or https) (default https), Customer S3-compatible endpoint signing region (default us-east-1; most S3-compatible stores accept any value), and Time series S3 path. It doesn't ask for Runtime AWS region for S3 access, because it uses the signing region instead.

  • azure: Azure Blob Storage. The installer asks for Time series Azure Storage account name, Time series Azure Storage account key, and Time series Azure Blob container/prefix path.

For aws-s3 and s3-compatible, the installer then asks for runtime S3 credentials if they aren't already configured. If the host already has an AWS credentials file (~/.aws/credentials), it first asks Use the existing AWS credentials file for runtime S3 access? [y/N]. Otherwise, or if you answer no, it asks for Runtime S3 AWS access key ID, Runtime S3 AWS secret access key, and Runtime S3 AWS session token (optional).

Support bundle upload to S3

Upload support bundles and support configs to S3? (? for help) [y/N]:

If you answer yes, the installer asks for Support upload S3 base path (an s3:// URI), and for the AWS region and runtime S3 credentials if they aren't already set. If you answer no, support bundles are stored locally.

Warning: Support bundles are used for system diagnostics and may contain sensitive information, so ensure S3 buckets follow your organization's security policies.

Outbound proxy

Enter outbound HTTP or HTTPS proxy URL if any. Hit Enter if none (? for help) :

Full proxy URL if required. Example: https://proxy.example.com:3128. Leave blank if no proxy is needed.

VM resource template

Select VM resource template (? for help)
 1. default
 2. default-highmem
 3. default-onemillion
 4. 832-onemillion
 5. default-fivemillion
 6. 3296-fivemillion
 7. 1k-gpu
Choose an option:

Select the template matching the provisioned hardware and expected ingestion rate. Use 1k-gpu for 1,000-GPU-class AI factory deployments; for that class, it replaces default-highmem. For single-node deployments, use the default template unless otherwise directed by NVIDIA support engineers.

Backup schedule

Enter time to run backup jobs.
  Format: <HH:MM>
  Current time on host is <HH:MM> (? for help) [00:00]:

Specify a low-activity time in the VM's configured timezone. If the VM runs UTC but the preferred backup time is 1 AM Pacific, enter the UTC equivalent.

Docker networking subnet

Enter an IP subnet for internal docker networking (? for help) [192.168.0.0/16]:

Accept the default unless it conflicts with on-premises subnets used by monitored devices. Must be at least /24.

Host time verification

Current time on host is <HH:MM> <timezone>.  Press Enter if time is accurate otherwise exit and configure correct time... (? for help) :

Verify the displayed time is correct. Any timezone is acceptable, but the clock must be accurate. If it's wrong, exit, fix the clock (see Time Synchronization), and run the installer again.

DNS verification

Make sure DNS is configured. Press Enter to continue... (? for help) :

Confirm DNS resolution works before continuing: nslookup google.com

Ingestion rate limits

For each data source that doesn't have a limit yet, in alphabetical order:

<datasource> rate limit
  Format: <count>/<duration>
    count: Integer
    duration: Integer[s/m/h]
 (? for help) [<default>]:

Data exceeding the rate limit is dropped to protect VM resources.

TSDB

Default Ingestion Rate Limit (per 1h)

raw_v1 appdata_v1 applog_v1

100,000

packet_v1 system_v1 syslog_v1 interface_v1 interface_nvlink_v1 histogram_v1 hft_summary_v1

300,000

On an upgrade, the default for interface_nvlink_v1 is the limit already set for interface_v1.

Data retention

For each data source (raw_v1, system_v1, packet_v1, syslog_v1, interface_v1, interface_nvlink_v1):

Enter data retention days for <datasource> [<default>]:

Set retention appropriate for the storage allocated in the chosen template.

TSDB

Default Data Retention (days)

raw_v1

120

syslog_v1

30

packet_v1

7

system_v1

10

interface_v1

10

interface_nvlink_v1

10

On an upgrade, the default for interface_nvlink_v1 is the retention already set for interface_v1.

High-frequency telemetry storage

Enter high-frequency telemetry retention hours for hft-db (? for help) [12]:
Enter the hft-db ingestion ceiling in datapoints per second (? for help) [4000000]:
Enter the hft-db rate limit grace, as a fraction of the ceiling (? for help) [0.05]:

Storage settings for high-frequency telemetry. The installer asks each one only while it isn't set, so an upgrade keeps the values you entered.

Setting

Default

Description

Retention hours

12

How long raw high-frequency telemetry is kept. Storage grows with the number of ports, the sample rate, and the retention time, so choose a period that fits the available disk. The configured disk budget can cause older samples to be removed sooner.

Ingestion ceiling

4000000

The number of incoming datapoints stored per second. Sustained excess datapoints are dropped and counted. Enter 0 to turn off the ceiling.

Rate limit grace

0.05

How far brief bursts can go above the ceiling, as a fraction from 0 to 1. For example, 0.05 allows five percent above the ceiling.

Setup name

Enter a unique name for this stack installation (? for help) :

One-word name that uniquely identifies this deployment. No spaces. Example: acme-prod

Manager URL

Enter URL of the manager VM on the platform stack (? for help) [https://<customer>.minerva.local]:

The URL used to access the NVIDIA Minerva UI. Used to construct permalinks in outgoing alert notifications. Example: https://minerva.nvidia.example.com

Enter the platform's reachable address. Don't keep the default unless it resolves in your network.

Backup destination

Enter backup destination (? for help)
 1. s3
 2. local
 3. all
Choose an option:

Automated backups of configs and databases are stored at this destination. s3 uses a customer-supplied S3 bucket. all enables both S3 and local backups.

If you choose s3 or all, the installer asks for Backup S3 path (an s3:// URI). It also asks for Runtime AWS region for backup S3 access and the runtime S3 credentials if they aren't already set.

Auditing

auditd isn't installed. Please make sure to install it using instructions in README. Press Enter to continue... (? for help) :

The installer shows this message if auditd is not installed. Install and configure it as described in auditd Configuration in Prepare Hosts for Docker Swarm before you continue.

Core dump verification

systemd-coredump isn't installed. Please make sure to install it using instructions in README. Press Enter to continue... (? for help) :

The installer shows this message if systemd-coredump isn't installed. Install it before you continue (see Prepare Hosts for Docker Swarm).

Logrotate verification

Make sure logrotate is configured for hourly rotate. Press Enter to continue... (? for help) :

The installer shows this message if /etc/cron.hourly/logrotate.syslog does not exist. See Configure Logrotate before you press Enter.

Bring Up the Stacks

From ~/minerva/ on the manager node, start the stacks:

./svm stack-up all

Stacks start in dependency order: infra → tsservice → backend → nvl → ui. Allow up to 10 minutes for tsservice to fully initialize.

Verify the Stacks

After ./svm stack-up all completes, verify that all services are running correctly:

cd ~/minerva
./svm stack-status

For a hybrid deployment, install the collectors next (see Collector Installation). For a unified deployment, continue with Complete Initial Setup.

Last updated: