Install on Multiple Nodes with High Availability

Purpose and Scope

You can configure a multi-node NVIDIA Minerva installation for high availability (HA). When a node fails, the services that can run on any node (the HA-capable services) restart automatically on another node. Configuration Rules covers which node runs which service, and Deployment Steps covers the installation.

The setup applies to a unified deployment and to the platform part of a hybrid deployment (collectors and platform). It uses one manager node (not a Docker Swarm with several managers) and several worker nodes. HA for the collectors isn't covered.

In scope. With enough HA nodes, sized for your ingestion rate, the deployment tolerates one node failure at a time. This holds only for the fixed set of HA nodes, where the HA-capable services run. You get this behavior by choosing the right service placements when you configure the deployment (see Configuration Rules and Deployment Steps). The sizing rule in Rule 3, the largest replica count of any HA-capable service plus one HA node, is what makes a single failure survivable. The other services (non-HA services) and the manager node stay on their own nodes and aren't relocated.

Out of scope: full HA. This setup is not full HA, in which the deployment tolerates the failure of any node, including the manager node or a collector node. Full HA and collector cluster HA are not covered.

How It Works

Docker Swarm restarts the tasks of a failed node on any other node that meets the service's placement constraint. Unless a service has a placement entry in deploy-cfg.yml, NVIDIA Minerva constrains it to node.role == manager, which matches only the manager node, so Swarm cannot move it. The HA setup changes this for selected services:

1. The nodes that run the HA-capable services (the HA nodes) get node labels that you choose.

2. Each selected service gets a placement constraint on one of those labels, so Swarm can run it on any node that has the label.

3. The HA-capable services that keep data (the data stores) keep their data folders on a shared NFS folder, so a data store can start on another HA node and open the same data. You provide the shared folder and mount it on every node.

Configuration Rules

These rules decide which node runs which service. Follow them before you edit deploy-cfg.yml.

Rule 1 — Split the Services into Two Groups

Every NVIDIA Minerva 1.1.0 service (91 in all) belongs to one of two groups:

• HA-capable services (47): the service can restart on another node and recover there. It can use local storage, but with shared storage (NFS), it comes back with its data after a short outage.

• Non-HA services (44): these services stay on the node where you place them. Either they depend on that node even with shared storage (a restart elsewhere corrupts data, data arrives at the node's address, or the service runs as a single instance), or their relocation hasn't been tested. Unlike the HA-capable services, they aren't relocated when their node fails.

Rule 2 — Decide Which Nodes Are HA Nodes and Which Run Non-HA Services

Divide the nodes you have into three roles:

• Manager node: runs the services that must stay with the manager, such as kafka, zookeeper, mgmt, ui, nginx, mysql*, fluentd, and lb. svm runs commands inside mysql* and kafka_acl_bootstrap on the manager, and the collector points at the manager's address, so these services cannot move.

• Non-HA nodes: the non-HA services run on the manager node or on dedicated nodes that aren't HA nodes. A node isn't an HA node when it has none of the labels that you use for the HA-capable services. The non-HA services don't need any HA labels.

• HA nodes: the worker nodes that you choose to run the HA-capable services. You choose their labels. Only HA-capable services run on them, and the labels of the HA-capable services exist only on these nodes.

Example layout, with one manager node, one non-HA node, and four HA nodes:

Rule 3 — Place HA-Capable Services Only on HA Nodes, with One Spare

1. Label the nodes. On the manager node, label the HA nodes. You can choose any labels, as long as you know which nodes are the HA nodes and you put the labels of the HA-capable services only on those nodes. Do not put any of these labels on the manager node or on other nodes that aren't HA nodes.

   Example with one label shared by several services:

   Bash

   docker node update --label-add worker=HAnode <ha-node-1-hostname>
   docker node update --label-add worker=HAnode <ha-node-2-hostname>
   docker node update --label-add worker=HAnode <ha-node-3-hostname>
   docker node update --label-add worker=HAnode <ha-node-4-hostname>
   

   Example with a label for one service (trans_metric_metrics on four HA nodes):

   Bash

   docker node update --label-add trans-metric-metrics=yes <ha-node-1-hostname>
   docker node update --label-add trans-metric-metrics=yes <ha-node-2-hostname>
   docker node update --label-add trans-metric-metrics=yes <ha-node-3-hostname>
   docker node update --label-add trans-metric-metrics=yes <ha-node-4-hostname>
   

2. Place each HA-capable service in deploy-cfg.yml. Add the entries to the placement: section (see deploy-cfg.yml Syntax Rules). Give each service the label key and value that you put on its nodes (sel and val), and set the replica count in a separate block. A service that isn't listed stays on the manager node. Each service has one placement entry: you can mix labels shared by several services with labels for a single service, but each service uses only one entry.

   YAML

   placement:
     trans_metric_metrics:
       sel: node.labels.trans-metric-metrics
       val: 'yes'
     middlemanager:
       sel: node.labels.worker
       val: HAnode
     # ... one entry for each HA capable service
   trans_metric_metrics:
     replicas: 3
   

   The non-HA services need no HA label, and they stay on the manager node unless you place them. To run a non-HA service on a dedicated node that isn't an HA node, give that node its own label (a key and value that you don't use for any HA-capable service), and give the service a placement entry with that label. Only services that don't need the manager can run elsewhere.

3. Keep a spare node. A failed service needs a free node to move to, so a service's label must be on at least its replica count plus one nodes. For example, trans_metric_metrics with three replicas needs its label on at least four nodes, as in the example above. One node failure then still leaves room for all three replicas. If all HA-capable services share one label, the minimum number of HA nodes is the largest replica count of any HA-capable service, plus one.

4. Size every HA node to take over from a failed node. After one HA node fails, the remaining HA nodes must run all the HA-capable services. Check CPU, memory, and, for middlemanager, free task slots.

5. Global services have no spare to move to. historical (HA-capable) and utility (non-HA) run one instance on every labeled node. When a node fails, one instance is lost, and Swarm does not replace it. This is acceptable: the setup runs with one instance fewer until the node returns. For historical, this relies on S3 or shared deep storage, and on enough disk space on the other nodes.

Requirements for HA-Capable Services

• Every volume of a data store (neo4j, consul, blobdb, anomalydb, anomalydb_ts, meta_device, meta_interface, and meta_custom) must be on shared storage.

• The NVL and provisioning services read ~/opt/minerva/etc/nvl, so ~/opt/minerva/etc must be on shared storage on every node, for example as a symbolic link to the shared folder. Create it before the installer runs on that node.

• The Druid tier needs spare task slots on the remaining HA nodes, Kafka retention longer than the recovery time, and S3 or shared deep storage.

• keycloak uses mysql_ui on the manager node. If the UI keeps returning HTTP 503 after ui, mysql_ui, and keycloak restart together, restart the UI: docker service update --force ui_ui.

Shared Storage Requirements

The data stores keep their data on the shared NFS folder, and their containers write to it as root. The shared folder must meet these requirements:

• NFS version: NFSv4.1 or later, with locking support.

• Export options: rw,sync,no_root_squash, limited to the cluster nodes. Root in the containers must be able to write to the folders and change their owners. With the default root_squash, the data stores fail to start.

• Mount options: a hard mount at the same path on every node, added to /etc/fstab so that it survives a reboot. A soft mount can corrupt data during a network problem. For example:

  <nfs-server>:/<export-path>  /nfs-share  nfs4  rw,hard,noatime,_netdev  0 0
  

Shared Storage Paths and Folders

/nfs-share is an example mount point. Every folder that a path override uses must exist before stack-up, because the installer doesn't create override paths. The folders must be owned by the install user, with the same UID and GID on every node. The NVL and provisioning services need no entry in the file below, because ~/opt/minerva/etc is a symbolic link to the shared folder /nfs-share/conf/etc.

This example ~/minerva/deploy-cfg.yml has the shared storage paths of the HA-capable services:

# ~/minerva/deploy-cfg.yml  (shared-storage paths of the HA capable services)
# Create this file before the installer runs. Folders listed here must exist before stack-up.
ac_py:
    volume:
        data: /nfs-share/data/service/ac-py
alarm_correlation:
    volume:
        data: /nfs-share/data/service/alarm-correlation
anomaly_detection:
    volume:
        data: /nfs-share/data/service/anomaly-detection
anomaly_detection_model:
    volume:
        data: /nfs-share/data/service/anomaly-detection-model
anomaly_detection_model_b:
    volume:
        data: /nfs-share/data/service/anomaly-detection-model-b
anomaly_detection_event_model:
    volume:
        data: /nfs-share/data/service/anomaly-detection-event-model
anomalydb:
    volume:
        data: /nfs-share/data/service/anomalydb
anomalydb_ts:
    volume:
        data: /nfs-share/data/service/anomalydb-ts
blobdb:
    volume:
        data: /nfs-share/data/service/blobdb
meta_device:
    volume:
        data: /nfs-share/data/service/meta-device
meta_interface:
    volume:
        data: /nfs-share/data/service/meta-interface
meta_custom:
    volume:
        data: /nfs-share/data/service/meta-custom
neo4j:
    volume:
        data: /nfs-share/data/service/neo4j
consul:
    volume:
        consul_data: /nfs-share/data/service/consul/data
        consul_config: /nfs-share/data/service/consul/config
otelcol_gateway_push:
    volume:
        data: /nfs-share/data/service/otel-collector
otelcol_processor:
    volume:
        data: /nfs-share/data/service/otel-collector
otelcol_internal:
    volume:
        data: /nfs-share/data/service/otel-collector
queryservice:
    volume:
        data: /nfs-share/data/service/queryservice
telemetry:
    volume:
        data: /nfs-share/data/service/telemetry
telemetry_internal:
    volume:
        data: /nfs-share/data/service/telemetry-internal
trans_metric:
    volume:
        data: /nfs-share/data/service/trans-metric
trans_metric_metrics:
    volume:
        data: /nfs-share/data/service/trans-metric-metrics
trans_metric_event:
    volume:
        data: /nfs-share/data/service/trans-metric-event
trans_metric_syslog:
    volume:
        data: /nfs-share/data/service/trans-metric-syslog
trans_metric_histogram:
    volume:
        data: /nfs-share/data/service/trans-metric-histogram
trans_topo:
    volume:
        data: /nfs-share/data/service/trans-topo
broker:
    volume:
        data_config: /nfs-share/data/service/tsservice/config
coordinator:
    volume:
        data_config: /nfs-share/data/service/tsservice/config
router:
    volume:
        data_config: /nfs-share/data/service/tsservice/config
historical:
    volume:
        data_config: /nfs-share/data/service/tsservice/config
middlemanager:
    volume:
        data_config: /nfs-share/data/service/tsservice/config
# utility is a Non-HA service but re-mounts the shared stores and the Druid config at its own paths:
# these must point at the same shared folders as the owning services.
utility:
    volume:
        neo4j_data: /nfs-share/data/service/neo4j
        meta_device_data: /nfs-share/data/service/meta-device
        meta_interface_data: /nfs-share/data/service/meta-interface
        meta_custom_data: /nfs-share/data/service/meta-custom
        middlemanager_config: /nfs-share/data/service/tsservice/config
# The NVL / provisioning services (nvl_*, provisioning_*, setting_mgr, retry_scheduler) need no entry here:
# they read ~/opt/minerva/etc/nvl, which is covered by the ~/opt/minerva/etc symlink to /nfs-share/conf/etc.

Create the folders once, as the install user, on any node that has the share mounted. The install user must be able to write to the share root, so that the new folders belong to that user:

mkdir -p /nfs-share/conf/etc /nfs-share/data/service/{ac-py,alarm-correlation,anomaly-detection,anomaly-detection-event-model,anomaly-detection-model,anomaly-detection-model-b,anomalydb,anomalydb-ts,blobdb,consul/data,consul/config,meta-custom,meta-device,meta-interface,neo4j,otel-collector,queryservice,telemetry,telemetry-internal,trans-metric,trans-metric-event,trans-metric-histogram,trans-metric-metrics,trans-metric-syslog,trans-topo,tsservice/config}
find /nfs-share -mindepth 2 -type d | wc -l   # expect 30 folders below /nfs-share
ls -ln /nfs-share/data/service                # the owner columns must be the install user's uid and gid

Rule 4 — Limit Replicas per Node

Without a limit, Swarm can place two replicas of a service on the same HA node, and after a node failure, the replacement can end up on the same node as a surviving replica. Set max_replicas_per_node: 1 for every HA-capable service with more than one replica, and keep each service's replica count at most the number of nodes with its label, minus one, so that the replacement always finds a free node.

svm picks up files named stack-<name>.override.yml in ~/minerva and merges them into that stack when it deploys it.

Each override file must declare the same version: as the stack files, or stack-up fails. To see the version the stack files use, run grep ^version ~/minerva/stack-*.yml.

# ~/minerva/stack-backend.override.yml
version: '3.8'
services:
  trans-metric-metrics:
    deploy:
      placement:
        max_replicas_per_node: 1
  trans-metric-syslog:
    deploy:
      placement:
        max_replicas_per_node: 1
  # ... the other multi-replica services of this stack

Use the service names exactly as the stack file spells them. After you start the stack (Step 8), check that the limit is applied, for example:

docker service inspect --format '{{.Spec.Name}} {{.Spec.TaskTemplate.Placement.MaxReplicas}}' backend_trans-metric-metrics

The expected value is 1. A value of 0 means that no limit is set. If you see 0, check the service name in the override file.

deploy-cfg.yml Syntax Rules

1. Use only one placement: section. If deploy-cfg.yml already has a placement: section, add your service entries to it. Otherwise, create one.

2. Use one value for each label key on a node. A node label key has one value on each node. If a node already has a label, for example worker=worker1 on the manager node, do not give that key another value on that node. A node can have any number of different label keys.

Deployment Steps

Run the steps in this order. Run svm and docker service commands on the manager node only. Paths use the defaults of an installation by a non-root user: ~/minerva for the deployment files, and ~/opt/minerva for service data.

Prerequisite: the shared NFS folder meets the Shared Storage Requirements (Rule 3), it's mounted at the same path on every node (for example, /nfs-share), and the install user can write to it.

Step 1 — Create the Shared Folders

The installer doesn't create the folders of the path overrides. Use the mkdir command in Shared Storage Paths and Folders (Rule 3). It includes /nfs-share/conf/etc, the target of the etc link in step 2.

Step 2 — Link etc to the Shared Folder on Every Node

The manager, utility, and the NVL and provisioning services read ~/opt/minerva/etc, so every node must see the same files. Run these commands on every node before the installer runs on that node:

mkdir -p ~/opt/minerva          # ~/opt may itself be a link to a data disk; create the folder first
ln -s /nfs-share/conf/etc ~/opt/minerva/etc
readlink -f ~/opt/minerva/etc   # expect /nfs-share/conf/etc

Step 3 — Create deploy-cfg.yml with the Shared Storage Paths

The path overrides must exist when the installer and svm first render the stacks. Create ~/minerva/deploy-cfg.yml from the example in Shared Storage Paths and Folders (Rule 3).

Step 4 — Run the Installer on the Manager, Then on Each Worker

Install NVIDIA Minerva on every node, and join the workers to the swarm.

1. On the manager node, run the installer with your platform settings, as for a single-node installation (./installer.minerva.<version> -a).

2. On the manager node, display the join token and address:

   Bash

   docker swarm join-token worker
   

3. On each worker, one at a time, run the installer with the join option. Wait until it finishes before you start the next worker:

   Bash

   ./installer.minerva.<version> -a -s "worker <token> <manager-addr>:2377"
   

After each worker has joined, check that your customized etc files are unchanged: global.json, backup.json, and setup.json. A worker installation rewrites only the .default and .new files.

md5sum ~/opt/minerva/etc/{global,backup,setup}.json

The installation also writes default files (*.json.new and similar) to the local default folders, not to the shared paths. The services start without them, so don't copy them.

Step 5 — Set the Node Labels

Mark the HA nodes with the labels you chose in Rule 2 and Rule 3. Put the labels of the HA-capable services only on the HA nodes.

docker node update --label-add worker=HAnode <ha-node-1-hostname>
# ... one command per HAnode ...   (the labels shown are examples)

Step 6 — Add the Placements, Replica Counts, and Replica Limits

Move the HA-capable services to the HA nodes, and keep one replica on each node.

1. In ~/minerva/deploy-cfg.yml, add the placement entries to the placement: section (Rule 3), and set the services' replica counts.

2. Create the stack-<name>.override.yml files with max_replicas_per_node: 1 for the services with more than one replica (Rule 4).

Step 7 — Apply the Tuning Files

Size the setup for its ingestion rate. The values depend on the rate and aren't covered in these steps. These are the files that deployments change:

The installer doesn't create ~/opt/minerva/etc/trans-metric.json. Its defaults are in the trans-metric image, and a file in etc overrides them. To change these settings, copy the defaults first, and then edit the copy:

docker run --rm --entrypoint cat augtera/trans-metric:current /etc/trans-metric.json > ~/opt/minerva/etc/trans-metric.json

The Druid configuration files historical.conf, middleManager.conf, and tsservice.conf don't exist until you create them. After the first start, the Druid services write templates named <file>.default to .../tsservice/config. They rewrite the templates on every start, so edit only your own copy. The Druid services make the folder owned by UID 1000, so you need sudo to create or edit a file there:

cd /nfs-share/data/service/tsservice/config
sudo cp historical.conf.default historical.conf
# edit historical.conf with sudo, then apply the change:
docker service update --force tsservice_historical

Use the same steps for middleManager.conf (service tsservice_middlemanager) and tsservice.conf. To apply a change to tsservice.conf, run ./svm stack-up tsservice.

.../tsservice/config is the Druid configuration folder (/nfs-share/data/service/tsservice/config in the shared layout). When you can, change files while the stack is stopped: stop all stacks, change the files, and then run ./svm stack-up all.

Step 8 — Start the Stack

Start all services, and then check that they reach their replica counts and run on the intended nodes.

cd ~/minerva
./svm stack-up all

After the Start

Install the license. The license-install command looks for a local license container. When the license replicas run on workers, the manager has none, so name the image:

LICENSE_IMAGE=augtera/license:current ~/minerva/bin/license-install <license-file>

Hybrid deployments: install the collectors. On the manager node, create the bootstrap bundle for each collector, copy it to the collector over a secure channel, and install the collector with it. The bundle contains a private key.

cd ~/minerva
./svm collector-bundle <collector-id>      # creates ~/minerva/collector-bootstrap-<collector-id>.tar.gz

On the collector, run the installer with MINERVA_DEPLOYMENT_TYPE=collector and MINERVA_COLLECTOR_BOOTSTRAP_BUNDLE=<path-to-the-bundle> in its environment.

The installer prepares the collector but doesn't start its services. When it finishes, start the collector stack on the collector:

cd ~/minerva
./svm stack-up all
./svm stack-status

Last updated: November 02, 2026