Install on Multiple Nodes

This guide covers the NVIDIA Minerva platform deployment on a multi-node Docker Swarm. Services are managed by the svm script and grouped into stacks. For the stacks that make up the platform, see Docker Swarm - Platform.

Before you begin, make sure every node meets the Resource Requirements and is prepared as described in Prepare Hosts for Docker Swarm.

The installer can run in three deployment types:

The manager node is installed first; worker nodes join the Swarm afterward. The following steps use the default-onemillion (3-node) template as the primary example. Adjust node counts and labels for other templates.

Install in this order:

1. Run the installer on the manager node.

2. Get the worker join token.

3. Run the installer on the worker nodes.

4. Assign node labels.

5. Configure deploy-cfg.yml.

6. Bring up the stacks.

Run the Installer on the Manager Node

1. Copy the installer package to the manager node and run it:

   chmod +x installer.minerva.<BUILD>
   ./installer.minerva.<BUILD>
   

NVIDIA Minerva has three installer versions. If you aren't using the standard installer, use its file name in the command above:

On ARM64 hosts, use the ARM64 builds of the bundled installers, which have arm64 in the name: installer.minerva.all.arm64.<BUILD> and installer.minerva.collector.arm64.<BUILD>. An installer run on a host of the other architecture stops, and its message names the installer to use.

2. Answer the installer prompts. Enter ? at any prompt for a description of that parameter. Answers are saved, so re-runs (upgrades) skip previously answered prompts. For what each prompt means, see Installer Prompts.

3. When the installer finishes, go to the installation directory. The software is installed under ~/minerva/:

   cd ~/minerva
   

4. For a hybrid installation, create a bootstrap bundle for each collector (see Create a Collector Bootstrap Bundle in Install Collectors for a Hybrid Deployment).

Installer Prompts

The prompts are listed in the order the installer asks them. Some appear only for certain deployment types, or only when a value isn't already set.

Deployment type

Select deployment type (? for help)
 1. platform
 2. collector
 3. unified
Choose an option:

Refer to Deployment Models for details on deployment types. For a hybrid installation, the Platform Stack must be fully installed before beginning the Collector Stack installation.

The collector installer (installer.minerva.collector.<BUILD>) doesn't ask this question. It always installs a collector deployment.

Container registry base path

Container registry base path (? for help) :

The Docker registry path the installer pulls images from, for example registry.example.com/<path>/. Do not include http:// or https://. If the registry requires authentication, run docker login on the node before you run the installer.

• NMC: Use nvcr.io.

• With a bundled installer (installer.minerva.all.<BUILD> or installer.minerva.collector.<BUILD>), or when you pass -p <registry-base>, the installer doesn't ask for a registry.

Platform Stack IP address or FQDN

Enter Platform Stack IP address or FQDN (? for help) [<host-name>]:

Platform deployments only. This is the address collectors use to reach the Platform Stack, and the platform's TLS certificate is generated for it. The default is the host name that hostname -f returns on this host. Enter an IP address or FQDN that every collector node can resolve and reach.

If you enter a host name, it must resolve through DNS. An entry in a collector's /etc/hosts isn't enough, because the collector's services run in containers that don't read it.

Kafka external IP address or FQDN

Kafka external FQDN or IP address for TLS listener (? for help) [<platform-address>]:

Platform deployments only. This is the address collectors use to reach Kafka on port 9093. The default is the Platform Stack address. Change it only if Kafka runs on a different node.

Additional platform TLS SANs

Additional platform TLS SANs comma-separated. Hit Enter if none (? for help) :

localhost, 127.0.0.1, and the platform and Kafka addresses are always included. Enter each SAN as DNS:<name> or IP:<address>, or as a bare name or address.

WJH gNMI dial-out username

WJH gNMI dial-out username. Hit Enter for the read-only default (? for help) [telemetry-ro]:

WJH gNMI dial-out password

WJH gNMI dial-out password. Hit Enter to leave unconfigured (? for help) :

Time-series storage type

Where would you like to save the time series data? (? for help)
 1. local
 2. aws-s3
 3. s3-compatible
 4. azure
Choose an option [local]:

Where NVIDIA Minerva stores time-series data.

• local: on the host. There are no further prompts.

• aws-s3: Amazon S3. The installer asks for Time series S3 path (an s3:// URI) and, if the AWS region isn't already set, Runtime AWS region for S3 access.

• s3-compatible: an S3-compatible object store that you provide, for example Ceph. The installer asks for Customer S3-compatible endpoint (host:port), Customer S3-compatible endpoint protocol (http or https) (default https), Customer S3-compatible endpoint signing region (default us-east-1; most S3-compatible stores accept any value), and Time series S3 path. It doesn't ask for Runtime AWS region for S3 access, because it uses the signing region instead.

• azure: Azure Blob Storage. The installer asks for Time series Azure Storage account name, Time series Azure Storage account key, and Time series Azure Blob container/prefix path.

For aws-s3 and s3-compatible, the installer then asks for runtime S3 credentials if they aren't already configured. If the host already has an AWS credentials file (~/.aws/credentials), it first asks Use the existing AWS credentials file for runtime S3 access? [y/N]. Otherwise, or if you answer no, it asks for Runtime S3 AWS access key ID, Runtime S3 AWS secret access key, and Runtime S3 AWS session token (optional).

Support bundle upload to S3

Upload support bundles and support configs to S3? (? for help) [y/N]:

If you answer yes, the installer asks for Support upload S3 base path (an s3:// URI), and for the AWS region and runtime S3 credentials if they aren't already set. If you answer no, support bundles are stored locally.

Outbound proxy

Enter outbound HTTP or HTTPS proxy URL if any. Hit Enter if none (? for help) :

Full proxy URL if required. Example: https://proxy.example.com:3128. Leave blank if no proxy is needed.

VM resource template

Select VM resource template (? for help)
 1. default
 2. default-highmem
 3. default-onemillion
 4. 832-onemillion
 5. default-fivemillion
 6. 3296-fivemillion
 7. 1k-gpu
Choose an option:

Select the template matching the provisioned hardware and expected ingestion rate. Use 1k-gpu for 1,000-GPU-class AI factory deployments; for that class, it replaces default-highmem. For single-node deployments, use the default template unless otherwise directed by NVIDIA support engineers.

Backup schedule

Enter time to run backup jobs.
  Format: <HH:MM>
  Current time on host is <HH:MM> (? for help) [00:00]:

Specify a low-activity time in the VM's configured timezone. If the VM runs UTC but the preferred backup time is 1 AM Pacific, enter the UTC equivalent.

Docker networking subnet

Enter an IP subnet for internal docker networking (? for help) [192.168.0.0/16]:

Accept the default unless it conflicts with on-premises subnets used by monitored devices. Must be at least /24.

Host time verification

Current time on host is <HH:MM> <timezone>.  Press Enter if time is accurate otherwise exit and configure correct time... (? for help) :

Verify the displayed time is correct. Any timezone is acceptable, but the clock must be accurate. If it's wrong, exit, fix the clock (see Time Synchronization), and run the installer again.

DNS verification

Make sure DNS is configured. Press Enter to continue... (? for help) :

Confirm DNS resolution works before continuing: nslookup google.com

Ingestion rate limits

For each data source that doesn't have a limit yet, in alphabetical order:

<datasource> rate limit
  Format: <count>/<duration>
    count: Integer
    duration: Integer[s/m/h]
 (? for help) [<default>]:

Data exceeding the rate limit is dropped to protect VM resources.

On an upgrade, the default for interface_nvlink_v1 is the limit already set for interface_v1.

Data retention

For each data source (raw_v1, system_v1, packet_v1, syslog_v1, interface_v1, interface_nvlink_v1):

Enter data retention days for <datasource> [<default>]:

Set retention appropriate for the storage allocated in the chosen template.

On an upgrade, the default for interface_nvlink_v1 is the retention already set for interface_v1.

High-frequency telemetry storage

Enter high-frequency telemetry retention hours for hft-db (? for help) [12]:
Enter the hft-db ingestion ceiling in datapoints per second (? for help) [4000000]:
Enter the hft-db rate limit grace, as a fraction of the ceiling (? for help) [0.05]:

Storage settings for high-frequency telemetry. The installer asks each one only while it isn't set, so an upgrade keeps the values you entered.

Setup name

Enter a unique name for this stack installation (? for help) :

One-word name that uniquely identifies this deployment. No spaces. Example: acme-prod

Manager URL

Enter URL of the manager VM on the platform stack (? for help) [https://<customer>.minerva.local]:

The URL used to access the NVIDIA Minerva UI. Used to construct permalinks in outgoing alert notifications. Example: https://minerva.nvidia.example.com

Enter the platform's reachable address. Don't keep the default unless it resolves in your network.

Backup destination

Enter backup destination (? for help)
 1. s3
 2. local
 3. all
Choose an option:

Automated backups of configs and databases are stored at this destination. s3 uses a customer-supplied S3 bucket. all enables both S3 and local backups.

If you choose s3 or all, the installer asks for Backup S3 path (an s3:// URI). It also asks for Runtime AWS region for backup S3 access and the runtime S3 credentials if they aren't already set.

Auditing

auditd isn't installed. Please make sure to install it using instructions in README. Press Enter to continue... (? for help) :

The installer shows this message if auditd is not installed. Install and configure it as described in auditd Configuration in Prepare Hosts for Docker Swarm before you continue.

Core dump verification

systemd-coredump isn't installed. Please make sure to install it using instructions in README. Press Enter to continue... (? for help) :

The installer shows this message if systemd-coredump isn't installed. Install it before you continue (see Prepare Hosts for Docker Swarm).

Logrotate verification

Make sure logrotate is configured for hourly rotate. Press Enter to continue... (? for help) :

The installer shows this message if /etc/cron.hourly/logrotate.syslog does not exist. See Configure Logrotate before you press Enter.

Get the Worker Join Token

1. On the manager node, retrieve the join token and swarm address for worker nodes:

   docker swarm join-token worker
   

2. Record the --token value and the <addr:port> (typically <manager-ip>:2377) from the output. You need both to run the installer on the worker nodes.

Run the Installer on the Worker Nodes

1. If you aren't using the full installer, log in to the container registry on the worker first (see Container Registry Authentication in Prepare Hosts for Docker Swarm).

2. On each worker node, run the installer in worker mode:

   chmod +x installer.minerva.<BUILD>
   ./installer.minerva.<BUILD> -p <registry-base> -s "worker <token> <addr:port>"
   

The worker installer downloads all Docker images and joins the node to the swarm. No interactive prompts are required.

Assign Node Labels

Node labels determine service placement and must be assigned before you bring up the stacks. On the manager node, assign the labels. For the default-onemillion (3-node) template:

# make note of node identifiers
docker node ls

# host1 = manager node
docker node update --label-add worker=worker1 host1

# host2 = historical storage and middle manager services
docker node update --label-add historical=yes host2
docker node update --label-add middlemanager=yes host2

# host3 = timeseries DB router, broker, and coordinator
docker node update --label-add tsservice_rbc=yes host3

Replace host1, host2, and host3 with the node identifiers shown by docker node ls.

Configure deploy-cfg.yml

deploy-cfg.yml under ~/minerva/, on the manager node, controls service placement and key operational parameters. NVIDIA support will provide a customized service placement configuration, as placement varies by hardware.

Do not edit deploy-cfg-default.yml — it is overwritten on every upgrade. All local configuration must go into deploy-cfg.yml.

Bring Up the Stacks

From ~/minerva/ on the manager node, start the stacks:

./svm stack-up all

Stacks start in dependency order: infra → tsservice → backend → nvl → ui. Allow up to 10 minutes for tsservice to fully initialize.

Verify the Stacks

After ./svm stack-up all completes, verify that all nodes and services are running correctly:

cd ~/minerva
./svm stack-status

Then verify node membership and service distribution:

minerva-cli support show-hosts
minerva-cli support show-distribution

show-hosts lists all nodes in the swarm. show-distribution shows which services are running on which node. Confirm that services are placed according to the labels and deploy-cfg.yml you configured earlier.

For a hybrid deployment, install the collectors next (see Collector Installation). For a unified deployment, continue with Complete Initial Setup.

Last updated: November 02, 2026