Networking Minerva User Guide

Install Collectors for a Hybrid Deployment

Install each collector after the Platform Stack is running, in this order:

  1. Create a collector bootstrap bundle on the platform.

  2. Run the collector installer.

  3. Configure deploy-cfg.yml.

  4. Add worker nodes (multi-node collectors only).

  5. Bring up the stacks.

Create a Collector Bootstrap Bundle

The bundle tells the collector how to reach the Platform Stack and gives it its identity. It contains the platform and Kafka addresses, the platform root CA, the collector's client certificate and key, and its collector ID. Create one bundle for each collector.

  1. Choose a collector ID: exactly 8 hexadecimal digits, unique for each collector. To generate one:

    python3 -c "import secrets; print(secrets.token_hex(4))"
    
  2. On the platform manager node, create the bundle:

    cd ~/minerva
    ./svm collector-bundle <collector-id>
    

    This writes ~/minerva/collector-bootstrap-<collector-id>.tar.gz.

    • All nodes of one collector install the same bundle and share its ID. A collector that needs its own identity needs its own bundle.

    • To reissue an existing bundle, add --force. Collectors already installed from the old bundle keep working. The reissued bundle has a new key pair, the same name, and the same CA; nothing is revoked.

Warning: Don't install a collector from the common bundle that the platform installation writes (~/minerva/collector-bootstrap.tar.gz). Its certificate names no collector, so the platform's Kafka doesn't authorize the collector's switch-connector. The installer accepts this bundle without a warning and exits successfully, and docker service ls still shows 1/1. The failure appears only in the switch-connector logs (Topic authorization failed, the ownership topic cannot be read ... NOT ready). Always use a named bundle.

Warning: The bundle includes sensitive credentials and certificates. Transmit it using a secure method that complies with your organization's information security policies.

  1. Copy the bundle and the collector installer to the collector's manager node:

    scp ~/minerva/collector-bootstrap-<collector-id>.tar.gz <user>@<collector-host>:~/
    scp installer.minerva.collector.<BUILD> <user>@<collector-host>:~/
    

    The bundle contains the collector's private key. On the collector's manager node, restrict it to the install user:

    chmod 600 ~/collector-bootstrap-<collector-id>.tar.gz
    

Run the Collector Installer

  1. On the collector's manager node, run the installer with your bundle:

    chmod +x installer.minerva.collector.<BUILD>
    COLLECTOR_BOOTSTRAP_BUNDLE=~/collector-bootstrap-<collector-id>.tar.gz ./installer.minerva.collector.<BUILD>
    

    The collector installer includes all collector images and doesn't ask for a deployment type. You can also use the standard installer (installer.minerva.<BUILD>): select collector when prompted, and make sure the node can reach the container registry.

    On ARM64 hosts, use installer.minerva.collector.arm64.<BUILD>.

  2. Answer the installer prompts (see Installer prompts). Enter ? at any prompt for help. The installer skips prompts that the bundle or a previous run already answered. With COLLECTOR_BOOTSTRAP_BUNDLE set, it prints Collector identity: <collector-id> and doesn't ask for the bundle path or the Kafka address.

Installer prompts

The prompts appear in this order:

  1. Collector bootstrap bundle file path: only when COLLECTOR_BOOTSTRAP_BUNDLE isn't set. A new installation fails without a bundle.

  2. Enter Platform Stack IP address or FQDN: the address the on-premises services use to reach the Platform Stack. The default comes from the bundle; press Enter to accept it.

  3. WJH gNMI dial-out username (default telemetry-ro) and password. Press Enter to leave them unconfigured.

  4. Upload support bundles and support configs to S3? [y/N]: if you answer yes, you're also prompted for the S3 base path.

  5. Enter a tenant name: leave blank. The value isn't used on a collector.

  6. Enter outbound HTTP or HTTPS proxy URL if any: the proxy that on-premises services use to reach the internet, for example https://proxy.example.com:3128. The host also needs a proxy in /etc/environment; see HTTP Proxy in Prepare Hosts for Docker Swarm. TCP connectivity from the collector to the platform on ports 443, 9093, 14317, 14321, 24317, and 29093 remains mandatory.

  7. Docker subnet (default 192.168.0.0/16), host time, DNS, auditd, systemd-coredump, and logrotate. For the Docker subnet, host time, DNS, auditd, and logrotate prompts, see Installer Prompts in Install on a Single Node. For systemd-coredump, see systemd-coredump Configuration in Prepare Hosts for Docker Swarm. The auditd and systemd-coredump prompts appear only if those packages are missing.

After the S3 prompt, the installer creates the Swarm and the Docker secrets and prints its progress before it continues with the tenant prompt.

If the installer prints error: crontab failed at the end, see User Setup in Prepare Hosts for Docker Swarm.

Configure deploy-cfg.yml

The installer creates and configures deploy-cfg.yml. Merge only approved local overrides into it, for example the proxy.

  • Don't change conf.ingestion.class or conf.collector.id. They hold the collector ID from the bundle. To change the ID, reinstall with a different bundle.

Don't edit deploy-cfg-default.yml. It's overwritten on every upgrade.

Add Worker Nodes (Multi-Node Collectors Only)

  1. On the collector's manager node, get the join token:

    docker swarm join-token worker
    
  2. On each worker node, run the installer with the token:

    chmod +x installer.minerva.collector.<BUILD>
    ./installer.minerva.collector.<BUILD> -s "worker <token> <addr:port>"
    

    Workers don't need the bundle and show no prompts. They get their secrets from the collector's manager node.

Bring Up the Stacks

  1. From ~/minerva/ on the collector's manager node, start the stacks and check their status:

    ./svm stack-up all
    ./svm stack-status
    
  2. Confirm that switch-connector is registered and ready. docker service ls showing 1/1 isn't enough: a collector with the wrong identity, or without Kafka access, stays Running but isn't ready.

    docker service ls | grep switch-connector
    docker service logs --since 2m backend_switch-connector 2>&1 | grep -E "registered with mgmt|started|assigned partitions|ERROR"
    

    A healthy collector logs switch-connector registered with mgmt, switch-connector started with "collectorId": "<collector-id>", and connector was assigned partitions, with no ERROR lines.

If the platform isn't reachable on port 9093, or the collector's identity isn't valid, switch-connector doesn't restart. It stays up and logs errors:

  • Port 9093 blocked: Message timed out, 1/1 brokers are down, the ownership topic cannot be read ... NOT ready, and could not publish its heartbeat. It recovers on its own once the port is open.

  • Wrong identity (common bundle): Topic authorization failed.

Reinstall or Move a Collector

  1. On the collector's manager node, stop the stacks: ./svm stack-down all. If the stacks are running, the installer refuses to continue (backend stack is running. Stop it by executing ./svm stack-down backend), but only after it has already imported the new certificate files.

  2. Run the installer with the new bundle, and then run ./svm stack-up all.

Installing with a bundle replaces the node's identity: root CA, client certificate, key, and collector ID. An upgrade without a bundle keeps the existing identity. When a collector moves to a new ID, the old ID's topics and registration stay on the platform.

For upgrades, see Upgrade NVIDIA Minerva on Docker Swarm.

Last updated: