Networking Minerva User Guide

Pre-install Requirements

The installation commands on this page require sudo. NVIDIA recommends creating a dedicated non-root account with passwordless sudo access and Docker group membership.

Supported Operating Systems

  • Ubuntu 24.04 LTS

Required Packages

Install the following on every node before running the installer.

On Ubuntu:

sudo apt-get update
sudo apt-get install -y \
  docker.io python3 python3-yaml ca-certificates openssl \
  logrotate rsyslog cron curl rsync git systemd-coredump auditd \
  tar gzip unzip

Install AWS CLI v2:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Docker version 20.10.23 or later is required. Version 27.5.0 is recommended. The installer verifies the Docker Engine version on every node and fails if any node reports lower than 20.10.23. If the distribution's default docker.io package is older than this, install Docker from the official Docker repository instead, or upgrade it before proceeding. Verify with:

docker version --format '{{.Server.Version}}'

Start and enable the daemon:

sudo systemctl enable --now docker

Disk Size

Minimum recommended values:

  • At least 20 GB free in $HOME

  • At least 50 GB for /var/lib/docker

  • ~/opt sized for the selected template and retention policy

For larger deployments, use the sizing recommended by NVIDIA support.

Data Directory Mount Options

If the data directory (/data or equivalent) is on its own XFS filesystem, mount it with:

noatime,logbsize=256k,largeio,inode64,nofail

These options are XFS-specific; a non-XFS filesystem should be reformatted as XFS rather than substituted with an equivalent, as no reliable ext4 equivalent exists for logbsize, largeio, or inode64.

Verify both the live mount and the persisted fstab entry — the two can drift independently:

findmnt -T /data
grep -v '^#' /etc/fstab | grep /data

nofail is a mount(8)/boot-time directive and will not appear in findmnt output even when correctly configured — check it in /etc/fstab only.

Swap

Disable swap on every node:

sudo swapoff -a

Remove or comment out any swap entry in /etc/fstab so it does not re-enable on reboot. Verify:

cat /proc/swaps

User Setup

Add the user that will run svm to the docker group:

sudo usermod -a -G docker <user>

Log out and back in for the change to take effect.

systemd-coredump Configuration

Route core dumps through systemd-coredump. These commands remove any hand-written core_pattern override and restore the package default:

sudo rm -f /etc/sysctl.d/50-coredump.conf
sudo apt-get install -y systemd-coredump
sudo sysctl --system
cat /proc/sys/kernel/core_pattern

The output must begin with | and contain systemd-coredump.

Core dumps land in /var/lib/systemd/coredump on the root volume, and the default limits allow up to 32 GB per dump. A large heap can produce a dump big enough to fill the root filesystem. Set explicit limits:

sudo tee /etc/systemd/coredump.conf > /dev/null <<'EOF'
[Coredump]
Compress=yes
ProcessSizeMax=8G
ExternalSizeMax=8G
MaxUse=10G
KeepFree=20G
EOF
sudo systemctl daemon-reload

KeepFree is the important setting — systemd refuses to write a dump if free space would fall below it, protecting the root filesystem. A process larger than ProcessSizeMax is skipped entirely, not truncated; raise the value only if the root volume has room for it. Verify:

coredumpctl list --no-pager

auditd Configuration

NVIDIA Minerva records execve system calls for audit trails. The installer warns if auditd is not configured.

Add the following audit rules:

sudo tee -a /etc/audit/rules.d/audit.rules > /dev/null << 'EOF'
-a exit,always -F arch=b32 -S execve -k auditcmd
-a exit,always -F arch=b64 -S execve -k auditcmd
EOF

sudo service auditd restart

Verify the rules are loaded:

sudo auditctl -l | grep auditcmd

sysctl I/O Tuning

Apply the following on every node:

sudo tee /etc/sysctl.d/70-minerva-io.conf > /dev/null << 'EOF'
vm.dirty_background_bytes = 268435456
vm.dirty_bytes            = 2147483648
vm.vfs_cache_pressure     = 50
vm.max_map_count          = 1048576
EOF
sudo sysctl --system

Verify:

sysctl vm.dirty_background_bytes vm.dirty_bytes vm.vfs_cache_pressure vm.max_map_count

Use sysctl --system, not sysctl -p against an individual file — the platform also ships a separate sysctl file using ratio-based dirty-page settings (vm.dirty_ratio/vm.dirty_background_ratio), which are mutually exclusive with the byte-based settings above. sysctl --system applies all files in the correct order and avoids silently zeroing out either set.

Network Requirements

Docker Subnet

NVIDIA Minerva uses 192.168.0.0/16 for internal Docker overlay networking by default. This subnet must not overlap with any on-premises subnets reachable by monitored devices — overlap will break gNMI collection. The subnet must be at least /24 (256 addresses).

Multi-NIC Hosts

On hosts with multiple network interfaces, specify the interface Docker Swarm should advertise by adding the following to /etc/docker/daemon.json on every node:

{
  "swarm-default-advertise-addr": "<interface-name-or-ip>"
}

Restart Docker after editing.

DNS Options

Check the DNS options on every node:

grep -E '^options' /etc/resolv.conf

If the output shows ndots:1, add this line inside the JSON object in /etc/docker/daemon.json (add a comma if other entries follow it):

"dns-opts": ["ndots:0"]

If the file also sets swarm-default-advertise-addr (see Multi-NIC Hosts), both settings go in the same object:

{
  "swarm-default-advertise-addr": "<interface-name-or-ip>",
  "dns-opts": ["ndots:0"]
}

Then restart Docker:

sudo systemctl restart docker

HTTP Access

Every node needs outbound HTTPS access to:

  • The configured container registry, unless using a bundled-image installer

  • S3 or Azure object storage, if used

  • Any configured support-bundle upload destination

Container Registry Authentication (NMC only)

If the provided installer is not the full installer (installer.minerva.all.<BUILD>), you must log in to the container registry on all nodes with your nvcr.io credentials.

docker login nvcr.io


HTTP Proxy

If outbound access requires a proxy, configure it on every node before running the installer. Edit /etc/environment:

http_proxy=http://PROXY_HOSTNAME:PROXY_PORT
https_proxy=http://PROXY_HOSTNAME:PROXY_PORT
no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com"
HTTP_PROXY=http://PROXY_HOSTNAME:PROXY_PORT
HTTPS_PROXY=http://PROXY_HOSTNAME:PROXY_PORT
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com"

Log out and back in after saving.

Multi-Node Swarm Port Requirements

Make sure the following ports are open for communication in a multi-node deployment:

On all nodes (managers + workers)

Port

Protocol

Purpose

7946

TCP/UDP

Swarm node communication

4789

UDP

Docker overlay network (VXLAN)

443

TCP

UI and API (HTTPS)

80

TCP

UI and API (HTTP)

On manager nodes only

Port

Protocol

Purpose

2377

TCP

Docker Swarm management

33352

UDP

Health-monitor agent report submission (from every platform node, including the manager itself)

On the node running the ingest task only

Port

Protocol

Purpose

4317

TCP

OTLP gRPC — device telemetry

4318

TCP

OTLP HTTP — device telemetry

4319

TCP

OTLP gRPC — NMX-T (NVLink) ingress

9091

TCP

Prometheus

57402

TCP

gNMI dial-out (WJH)

514

UDP

Syslog

4739

UDP

IPFIX

Hybrid Deployments: Collector to Platform Ports

In a hybrid deployment, open the ports listed under Inbound to Platform node (from collectors) in Collector Installation. Make sure 14317 and 24317 are open: collectors send data-plane metrics (14317) and their own health metrics and logs (24317) over them.

Loopback Routing for Self-Addressed Traffic

Some host-hardening firewall baselines include an anti-spoof rule that rejects traffic addressed to 127.0.0.0/8 when it arrives on a non-loopback interface. On a Docker Swarm node, traffic that round-trips through the ingress routing mesh (for example, a container reaching another local service via its published port) legitimately arrives back at the host over a bridge interface addressed to 127.0.0.1, not over lo.

If an anti-spoof rule of this kind is positioned ahead of the RELATED,ESTABLISHED accept rule in the host's INPUT chain, it silently drops this return traffic. The symptom is a service or health agent that appears to start normally but never completes a connection to a local, loopback-addressed endpoint, with no corresponding error in the host firewall log.

Before deploying, confirm whether such a rule exists:

sudo iptables -S INPUT | grep -i '127.0.0.0/8'

If present and positioned before the established-connection accept rule, add an explicit accept rule ahead of it for established/related traffic arriving on the Docker bridge interface:

sudo iptables -I INPUT 1 -i docker_gwbridge -d 127.0.0.0/8 \
  -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Persist the change according to the host's standard firewall-persistence mechanism so it survives a reboot.

Configure Logrotate

Cap the systemd journal so it cannot independently fill the disk:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/limits.conf >/dev/null <<'EOF'
[Journal]
SystemMaxUse=1G
SystemMaxFileSize=100M
SystemKeepFree=2G
EOF
sudo systemctl restart systemd-journald

Verify:

journalctl --disk-usage

NVIDIA Minerva generates high-volume syslog traffic. Ensure hourly log rotation is configured for /var/log/syslog to prevent disk exhaustion. This must be done for all nodes in a multi-node deployment (or just the single node for a single node deployment).

Create /etc/cron.hourly/logrotate.syslog if it does not exist:

sudo tee /etc/cron.hourly/logrotate.syslog >/dev/null << 'EOF'
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.d/rsyslog
EOF

sudo chmod 755 /etc/cron.hourly/logrotate.syslog

Add a maxsize directive to /etc/logrotate.d/rsyslog to cap individual log file size (for example, maxsize 500M). Before adding the directive, check for an existing maxsize setting to avoid duplicate entries.

If you back up the existing rsyslog config, place the backup outside /etc/logrotate.d/. Files in that directory are processed by logrotate and a misplaced backup will cause duplicate log rotation entries.

Copy the NVIDIA Minerva Installer to the Home Directory

This guide assumes the NVIDIA Minerva installer is located in the home directory of the installation user.

Manager SSH Access

Some svm operations use SSH/SCP from the manager to worker nodes. For multi-node installations, the manager installation user must be able to SSH to each worker without a password.

From the manager, create an SSH key if the installation user does not already have one:

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519

Copy the manager public key to each worker:

ssh-copy-id <worker-user>@<worker-host>

If ssh-copy-id is not available, append the manager public key to the worker user's ~/.ssh/authorized_keys.

Verify access from the manager:

ssh <worker-user>@<worker-host> hostname

Next: Single-node Installation | Multi-Node Installation

Last updated: