Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

Back Up Third-Party Configurations

This procedure collects third-party configuration files from your NVIDIA Minerva installation and packages them into a timestamped archive. The following configuration files are included:

  • Docker Swarm: Prometheus, syslog-ng, Vector, tsservice, OTel/DMT gateway, Nginx, and Fluentd

  • Kubernetes MinervaInfra and MinervaApp custom resources

All commands run in bash. Run them on a Linux host with bash, rsync, docker, and tar available for a Docker Swarm installation, or kubectl for a Kubernetes installation.

Run the entire procedure in a single shell session; later steps depend on environment variables exported by earlier ones. If your login shell is not bash (csh, tcsh, zsh, or dash, for example), run bash first and use it for the entire procedure.

Scope

This procedure checks for configuration files on disk; services do not need to be running. If no files are found for a given service, that step records skip: <reason> in probe.out and continues.

The archive may capture OS-package default files. For example, /etc/prometheus/prometheus.yml and /etc/syslog-ng/syslog-ng.conf exist on any Ubuntu host with those packages installed, even if those services were never part of your NVIDIA Minerva deployment. Before restoring, verify that the captured configs reflect your actual deployment.

For each service, only a specific set of file types is captured: Prometheus is limited to *.yml, *.yaml, and *.rules files under its config directory; Nginx and Fluentd are limited to *.conf files. Non-.conf includes and Prometheus rule files at absolute paths outside the resolved config directory are not captured.

Before You Begin: Determine Your Deployment Type

If you already know whether your installation is Docker Swarm or Kubernetes, skip to the relevant section. Otherwise, SSH to the NVIDIA Minerva host:

ssh <minerva-user>@<minerva-host>

If your login shell is not bash (csh, tcsh, zsh, dash, etc.), switch to bash and stay in it for the rest of the procedure. Then run the following probe to detect the deployment type:

if command -v docker >/dev/null 2>&1 && docker info 2>/dev/null | grep -q '^ *Swarm: active' && docker service ls --format '{{.Name}}' 2>/dev/null | grep -qE '^(backend_|infra_|tsservice_|ui_)'; then echo "Deployment: Docker Swarm -- follow the Docker Swarm section."; elif command -v kubectl >/dev/null 2>&1 && kubectl get crd minervainfras.minerva.minerva.nvidia.com >/dev/null 2>&1; then echo "Deployment: Kubernetes -- follow the Kubernetes section (context=$(kubectl config current-context))."; else echo "No active Minerva deployment detected from this shell. Confirm you're on the Minerva install host."; fi

Docker Swarm Deployment

The following table maps each service to its source and destination under ${BACKUP_ROOT}. Missing optional sources are recorded as skip: <reason> in ${BACKUP_ROOT}/probe.out and are not failures.

Item

Source

Destination

Prometheus client

Resolved Prometheus config directory from running prometheus --config.file; fallback /etc/prometheus/

${BACKUP_ROOT}/prometheus-client/

syslog-ng

Resolved syslog-ng config directory from running syslog-ng -f / --cfgfile; fallback /etc/syslog-ng/

${BACKUP_ROOT}/syslog-ng/

Vector

${MINERVA_ROOT}/var/vector/ when *.toml* files exist

${BACKUP_ROOT}/vector/

Druid / tsservice overrides

${MINERVA_ROOT}/var/tsservice/config/ when non-empty

${BACKUP_ROOT}/tsservice/

OTel / DMT gateway

~/minerva/dmt-gateway.yaml; fallback ${MINERVA_ROOT}/var/otel-collector/gateway.yaml

${BACKUP_ROOT}/otel-dmt-gateway

Nginx runtime edits

${MINERVA_ROOT}/var/nginx/*.conf

${BACKUP_ROOT}/nginx

Fluentd drop-ins

${MINERVA_ROOT}/var/fluentd/*.conf

${BACKUP_ROOT}/fluentd

  1. SSH to the manager host:

    ssh <minerva-user>@<swarm-manager-host>

  2. Prepare the backup directory. MINERVA_ROOT defaults to ${HOME%/}/opt/minerva (the standard non-root installation root). If your installation uses a different root path, export MINERVA_ROOT=/path/to/install before running this step.

    export MINERVA_ROOT="${MINERVA_ROOT:-${HOME%/}/opt/minerva}"
export TS=$(date -u +%Y%m%dT%H%M%SZ)
export BACKUP_ROOT="${HOME%/}/manual-backups/third-party-config/${TS}"
RSYNC_EXCLUDES=(--exclude='*.key' --exclude='*.pem' --exclude='*.crt' --exclude='*.pfx' --exclude='*secret*' --exclude='*.log' --exclude='data/' --exclude='segments/')
mkdir -p "${BACKUP_ROOT}"/{context,prometheus-client,syslog-ng,vector,tsservice,otel-dmt-gateway,nginx,fluentd}
: > "${BACKUP_ROOT}/probe.out"
echo "MINERVA_ROOT=${MINERVA_ROOT}"
echo "BACKUP_ROOT=${BACKUP_ROOT}"

  3. Record the environment context:

    {
  echo "=== date ===";        date -u
  echo "=== hostname ===";    hostname -f
  echo "=== docker nodes ==="; docker node ls
  echo "=== docker stacks ==="; docker stack ls
  echo "=== docker services ==="; docker service ls
} | tee "${BACKUP_ROOT}/context/swarm-context.txt"

  4. Copy the configuration files. Run each of the following blocks in sequence.

    Prometheus client

    ITEM=prometheus-client
DEST="${BACKUP_ROOT}/${ITEM}"
PROM_PID=$(pgrep -x prometheus | head -1)
PROM_CFG=
if [ -n "$PROM_PID" ]; then
  PROM_CFG=$(tr '\0' '\n' < /proc/$PROM_PID/cmdline | awk '
    /^--config\.file=/ { sub(/^--config\.file=/, ""); print; exit }
    /^--config\.file$/ { f=1; next }
    f { print; exit }')
fi
[ -z "$PROM_CFG" ] && PROM_CFG=/etc/prometheus/prometheus.yml
RUN_NOTE="${PROM_PID:+ (running pid=${PROM_PID})}"; RUN_NOTE="${RUN_NOTE:- (not running)}"
if [ ! -f "$PROM_CFG" ]; then
  echo "skip: prometheus cfg not found at $PROM_CFG" | tee -a "${BACKUP_ROOT}/probe.out"
else
  PROM_DIR=$(dirname "$PROM_CFG")
  rsync -a "${RSYNC_EXCLUDES[@]}" \
    --include='*/' --include='*.yml' --include='*.yaml' --include='*.rules' --exclude='*' \
    "${PROM_DIR}/" "${DEST}/"
  cp "$PROM_CFG" "${DEST}/"
  echo "copied: prometheus-client from ${PROM_DIR} (cfg=${PROM_CFG})${RUN_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    syslog-ng

    ITEM=syslog-ng
DEST="${BACKUP_ROOT}/${ITEM}"
SLNG_PID=$(pgrep -x syslog-ng | head -1)
SLNG_CFG=
if [ -n "$SLNG_PID" ]; then
  SLNG_CFG=$(tr '\0' '\n' < /proc/$SLNG_PID/cmdline | awk '
    /^--cfgfile=/ { sub(/^--cfgfile=/, ""); print; exit }
    /^-f$/ { f=1; next }
    f { print; exit }')
fi
[ -z "$SLNG_CFG" ] && SLNG_CFG=/etc/syslog-ng/syslog-ng.conf
RUN_NOTE="${SLNG_PID:+ (running pid=${SLNG_PID})}"; RUN_NOTE="${RUN_NOTE:- (not running)}"
if [ ! -f "$SLNG_CFG" ]; then
  echo "skip: syslog-ng cfg not found at $SLNG_CFG" | tee -a "${BACKUP_ROOT}/probe.out"
else
  SLNG_DIR=$(dirname "$SLNG_CFG")
  rsync -a "${RSYNC_EXCLUDES[@]}" "${SLNG_DIR}/" "${DEST}/"
  echo "copied: syslog-ng from ${SLNG_DIR} (cfg=${SLNG_CFG})${RUN_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    Vector

    ITEM=vector
DEST="${BACKUP_ROOT}/${ITEM}"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)vector$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/vector" ]; then
  echo "skip: ${MINERVA_ROOT}/var/vector not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/vector" -type f -name '*.toml*' -print -quit)" ]; then
  echo "skip: no vector .toml files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/vector/" "${DEST}/"
  echo "copied: vector from ${MINERVA_ROOT}/var/vector${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    Druid/tsservice runtime overrides

    ITEM=tsservice
DEST="${BACKUP_ROOT}/${ITEM}"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)tsservice_' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/tsservice/config" ]; then
  echo "skip: ${MINERVA_ROOT}/var/tsservice/config not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/tsservice/config" -type f -print -quit)" ]; then
  echo "skip: tsservice config dir is empty" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/tsservice/config/" "${DEST}/"
  echo "copied: tsservice from ${MINERVA_ROOT}/var/tsservice/config${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    OTel/DMT gateway — prefers ~/minerva/dmt-gateway.yaml; falls back to ${MINERVA_ROOT}/var/otel-collector/gateway.yaml, recording fallback: when used.

    ITEM=otel-dmt-gateway
DEST="${BACKUP_ROOT}/${ITEM}"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E 'otel-collector-gateway$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ -f ~/minerva/dmt-gateway.yaml ]; then
  rsync -a "${RSYNC_EXCLUDES[@]}" ~/minerva/dmt-gateway.yaml "${DEST}/"
  echo "copied: otel-dmt-gateway canonical ~/minerva/dmt-gateway.yaml${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -f "${MINERVA_ROOT}/var/otel-collector/gateway.yaml" ]; then
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/otel-collector/gateway.yaml" "${DEST}/gateway.yaml"
  echo "fallback: otel-dmt-gateway effective ${MINERVA_ROOT}/var/otel-collector/gateway.yaml (canonical missing)${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
else
  echo "skip: no dmt-gateway.yaml or effective gateway.yaml found" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    Nginx runtime edits — covers the proxy (nginx), data-plane LB (nginx-data), and on-premises variant (nginx-data-onprem). You do not need to know which services the installation runs.

    ITEM=nginx
DEST="${BACKUP_ROOT}/${ITEM}"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)nginx(-[a-z0-9-]+)?$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/nginx" ]; then
  echo "skip: ${MINERVA_ROOT}/var/nginx not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/nginx" -maxdepth 1 -type f -name '*.conf' -print -quit)" ]; then
  echo "skip: no nginx *.conf files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" --include='*.conf' --include='*/' --exclude='*' \
    "${MINERVA_ROOT}/var/nginx/" "${DEST}/"
  echo "copied: nginx from ${MINERVA_ROOT}/var/nginx${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

    Fluentd drop-ins

    ITEM=fluentd
DEST="${BACKUP_ROOT}/${ITEM}"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)fluentd$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ -z "$(find "${MINERVA_ROOT}/var/fluentd" -maxdepth 1 -type f -name '*.conf' -print -quit 2>/dev/null)" ]; then
  echo "skip: no fluentd drop-in *.conf files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" --include='*.conf' --exclude='*' \
    "${MINERVA_ROOT}/var/fluentd/" "${DEST}/"
  echo "copied: fluentd from ${MINERVA_ROOT}/var/fluentd${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

  5. Create the manifest:

    find "${BACKUP_ROOT}" -type f -print | sort > "${BACKUP_ROOT}/manifest.files"
if command -v sha256sum >/dev/null; then
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 sha256sum > "${BACKUP_ROOT}/manifest.sha256"
else
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 shasum -a 256 > "${BACKUP_ROOT}/manifest.sha256"
fi

  6. Package the archive:

    cd "$(dirname "${BACKUP_ROOT}")"
tar czf "third-party-config-${TS}.tar.gz" "$(basename "${BACKUP_ROOT}")"
ls -lh "third-party-config-${TS}.tar.gz"

    Store the archive in your backup set.

  7. Print the summary:

    ARCHIVE="$(dirname ${BACKUP_ROOT})/third-party-config-${TS}.tar.gz"
echo
echo "================ Third-Party Config Backup Summary ================"
echo "Timestamp:    ${TS}"
echo "Host:         $(hostname -f)"
echo "Backup dir:   ${BACKUP_ROOT}"
echo "Archive:      ${ARCHIVE}"
echo "Archive size: $(du -h "${ARCHIVE}" | cut -f1)"
echo "File count:   $(wc -l < "${BACKUP_ROOT}/manifest.files")"
echo
echo "Per-item result:"
sed 's/^/  /' "${BACKUP_ROOT}/probe.out"
echo "==================================================================="

    The summary lists every item as copied, skip: <reason>, or fallback. Anything else indicates an error during the procedure.

Kubernetes Deployment

You need a host with kubectl configured against your cluster and read access to the minerva.minerva.nvidia.com CRDs.

  1. Confirm your cluster context:

    kubectl config current-context
kubectl auth can-i get minervainfras.minerva.minerva.nvidia.com -A
kubectl auth can-i get minervaapps.minerva.minerva.nvidia.com -A

  2. Create the backup directory:

    export TS=$(date -u +%Y%m%dT%H%M%SZ)
export BACKUP_ROOT="${HOME%/}/manual-backups/third-party-config-k8s/${TS}"
mkdir -p "${BACKUP_ROOT}/custom-resources" "${BACKUP_ROOT}/context"
: > "${BACKUP_ROOT}/probe.out"
echo "BACKUP_ROOT=${BACKUP_ROOT}"

  3. Record the cluster context:

    {
  echo "=== date ===";          date -u
  echo "=== kubectl version ==="; kubectl version
  echo "=== cluster-info ===";  kubectl cluster-info
  echo "=== nodes ===";         kubectl get nodes -o wide
  echo "=== minervainfras ==="; kubectl get minervainfras -A
  echo "=== minervaapps ===";   kubectl get minervaapps -A
} > "${BACKUP_ROOT}/context/k8s-context.txt"

  4. Export custom resources:

    for KIND in minervainfras minervaapps; do
  if ! kubectl get crd "${KIND}.minerva.minerva.nvidia.com" >/dev/null 2>&1; then
    echo "skip: CRD ${KIND} not installed" | tee -a "${BACKUP_ROOT}/probe.out"
    continue
  fi
  if [ -z "$(kubectl get "${KIND}" -A -o name 2>/dev/null | head -1)" ]; then
    echo "skip: no ${KIND} resources found" | tee -a "${BACKUP_ROOT}/probe.out"
    continue
  fi
  kubectl get "${KIND}" -A -o yaml > "${BACKUP_ROOT}/custom-resources/${KIND}.yaml"
  echo "copied: ${KIND}" | tee -a "${BACKUP_ROOT}/probe.out"
done

  5. Create the manifest and archive:

    find "${BACKUP_ROOT}" -type f -print | sort > "${BACKUP_ROOT}/manifest.files"
if command -v sha256sum >/dev/null; then
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 sha256sum > "${BACKUP_ROOT}/manifest.sha256"
else
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 shasum -a 256 > "${BACKUP_ROOT}/manifest.sha256"
fi
cd "$(dirname "${BACKUP_ROOT}")"
tar czf "third-party-config-k8s-${TS}.tar.gz" "$(basename "${BACKUP_ROOT}")"
ls -lh "third-party-config-k8s-${TS}.tar.gz"

  6. Print the summary:

    ARCHIVE="$(dirname ${BACKUP_ROOT})/third-party-config-k8s-${TS}.tar.gz"
echo
echo "============ Third-Party Config Backup (Kubernetes) Summary ============"
echo "Timestamp:    ${TS}"
echo "Context:      $(kubectl config current-context)"
echo "Backup dir:   ${BACKUP_ROOT}"
echo "Archive:      ${ARCHIVE}"
echo "Archive size: $(du -h "${ARCHIVE}" | cut -f1)"
echo "File count:   $(wc -l < "${BACKUP_ROOT}/manifest.files")"
echo
echo "Per-item result:"
sed 's/^/  /' "${BACKUP_ROOT}/probe.out"
echo "========================================================================"

Verify the Archive

Run the following in the same shell session after completing the deployment steps. It resolves the archive path automatically, runs five checks, and prints OK or FAIL per check. A typical deployment archive is under 50 MB; if the archive is significantly larger, investigate before including it in your backup set.

if [ -z "${BACKUP_ROOT:-}" ] || [ -z "${TS:-}" ]; then
  echo "FAIL: BACKUP_ROOT or TS is not set in this shell."
  echo "      Verify must run in the same shell that ran the prepare step."
elif [ ! -d "${BACKUP_ROOT}" ]; then
  echo "FAIL: ${BACKUP_ROOT} does not exist."
else
  ARCHIVE=$(ls "$(dirname "${BACKUP_ROOT}")"/*${TS}*.tar.gz 2>/dev/null | head -1)
  if [ -z "${ARCHIVE}" ] || [ ! -f "${ARCHIVE}" ]; then
    echo "FAIL: no archive matching TS=${TS} under $(dirname "${BACKUP_ROOT}")"
  else
    export ARCHIVE
    echo "ARCHIVE=${ARCHIVE}"
    ls -lh "${ARCHIVE}"
    echo "--- probe.out ---"
    cat "${BACKUP_ROOT}/probe.out"
    echo "--- bulk data (must be empty) ---"
    tar tzf "${ARCHIVE}" | grep -E '(/data/|/segments/|\.log$)' && echo FAIL || echo OK
    echo "--- manifest integrity (every file must report OK) ---"
    ( cd "${BACKUP_ROOT}" && { sha256sum -c manifest.sha256 2>/dev/null || shasum -a 256 -c manifest.sha256; } ) | grep -v ': OK$' || echo all-OK
  fi
fi

Full End-to-End Commands

The following section contains all Docker Swarm and Kubernetes commands concatenated for copy-paste in a single session.

Docker Swarm 

# 1. Prepare
export MINERVA_ROOT="${MINERVA_ROOT:-${HOME%/}/opt/minerva}"
export TS=$(date -u +%Y%m%dT%H%M%SZ)
export BACKUP_ROOT="${HOME%/}/manual-backups/third-party-config/${TS}"
RSYNC_EXCLUDES=(--exclude='*.key' --exclude='*.pem' --exclude='*.crt' --exclude='*.pfx' --exclude='*secret*' --exclude='*.log' --exclude='data/' --exclude='segments/')
mkdir -p "${BACKUP_ROOT}"/{context,prometheus-client,syslog-ng,vector,tsservice,otel-dmt-gateway,nginx,fluentd}
: > "${BACKUP_ROOT}/probe.out"

# 2. Context
{
  echo "=== date ===";          date -u
  echo "=== hostname ===";      hostname -f
  echo "=== docker nodes ===";  docker node ls
  echo "=== docker stacks ==="; docker stack ls
  echo "=== docker services ==="; docker service ls
} | tee "${BACKUP_ROOT}/context/swarm-context.txt"

# 3a. Prometheus client
DEST="${BACKUP_ROOT}/prometheus-client"
PROM_PID=$(pgrep -x prometheus | head -1)
PROM_CFG=
if [ -n "$PROM_PID" ]; then
  PROM_CFG=$(tr '\0' '\n' < /proc/$PROM_PID/cmdline | awk '
    /^--config\.file=/ { sub(/^--config\.file=/, ""); print; exit }
    /^--config\.file$/ { f=1; next }
    f { print; exit }')
fi
[ -z "$PROM_CFG" ] && PROM_CFG=/etc/prometheus/prometheus.yml
RUN_NOTE="${PROM_PID:+ (running pid=${PROM_PID})}"; RUN_NOTE="${RUN_NOTE:- (not running)}"
if [ ! -f "$PROM_CFG" ]; then
  echo "skip: prometheus cfg not found at $PROM_CFG" | tee -a "${BACKUP_ROOT}/probe.out"
else
  PROM_DIR=$(dirname "$PROM_CFG")
  rsync -a "${RSYNC_EXCLUDES[@]}" \
    --include='*/' --include='*.yml' --include='*.yaml' --include='*.rules' --exclude='*' \
    "${PROM_DIR}/" "${DEST}/"
  cp "$PROM_CFG" "${DEST}/"
  echo "copied: prometheus-client from ${PROM_DIR} (cfg=${PROM_CFG})${RUN_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3b. syslog-ng
DEST="${BACKUP_ROOT}/syslog-ng"
SLNG_PID=$(pgrep -x syslog-ng | head -1)
SLNG_CFG=
if [ -n "$SLNG_PID" ]; then
  SLNG_CFG=$(tr '\0' '\n' < /proc/$SLNG_PID/cmdline | awk '
    /^--cfgfile=/ { sub(/^--cfgfile=/, ""); print; exit }
    /^-f$/ { f=1; next }
    f { print; exit }')
fi
[ -z "$SLNG_CFG" ] && SLNG_CFG=/etc/syslog-ng/syslog-ng.conf
RUN_NOTE="${SLNG_PID:+ (running pid=${SLNG_PID})}"; RUN_NOTE="${RUN_NOTE:- (not running)}"
if [ ! -f "$SLNG_CFG" ]; then
  echo "skip: syslog-ng cfg not found at $SLNG_CFG" | tee -a "${BACKUP_ROOT}/probe.out"
else
  SLNG_DIR=$(dirname "$SLNG_CFG")
  rsync -a "${RSYNC_EXCLUDES[@]}" "${SLNG_DIR}/" "${DEST}/"
  echo "copied: syslog-ng from ${SLNG_DIR} (cfg=${SLNG_CFG})${RUN_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3c. Vector
DEST="${BACKUP_ROOT}/vector"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)vector$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/vector" ]; then
  echo "skip: ${MINERVA_ROOT}/var/vector not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/vector" -type f -name '*.toml*' -print -quit)" ]; then
  echo "skip: no vector .toml files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/vector/" "${DEST}/"
  echo "copied: vector from ${MINERVA_ROOT}/var/vector${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3d. tsservice
DEST="${BACKUP_ROOT}/tsservice"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)tsservice_' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/tsservice/config" ]; then
  echo "skip: ${MINERVA_ROOT}/var/tsservice/config not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/tsservice/config" -type f -print -quit)" ]; then
  echo "skip: tsservice config dir is empty" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/tsservice/config/" "${DEST}/"
  echo "copied: tsservice from ${MINERVA_ROOT}/var/tsservice/config${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3e. OTel / DMT gateway
DEST="${BACKUP_ROOT}/otel-dmt-gateway"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E 'otel-collector-gateway$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ -f ~/minerva/dmt-gateway.yaml ]; then
  rsync -a "${RSYNC_EXCLUDES[@]}" ~/minerva/dmt-gateway.yaml "${DEST}/"
  echo "copied: otel-dmt-gateway canonical ~/minerva/dmt-gateway.yaml${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -f "${MINERVA_ROOT}/var/otel-collector/gateway.yaml" ]; then
  rsync -a "${RSYNC_EXCLUDES[@]}" "${MINERVA_ROOT}/var/otel-collector/gateway.yaml" "${DEST}/gateway.yaml"
  echo "fallback: otel-dmt-gateway effective ${MINERVA_ROOT}/var/otel-collector/gateway.yaml (canonical missing)${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
else
  echo "skip: no dmt-gateway.yaml or effective gateway.yaml found" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3f. Nginx
DEST="${BACKUP_ROOT}/nginx"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)nginx(-[a-z0-9-]+)?$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ ! -d "${MINERVA_ROOT}/var/nginx" ]; then
  echo "skip: ${MINERVA_ROOT}/var/nginx not present" | tee -a "${BACKUP_ROOT}/probe.out"
elif [ -z "$(find "${MINERVA_ROOT}/var/nginx" -maxdepth 1 -type f -name '*.conf' -print -quit)" ]; then
  echo "skip: no nginx *.conf files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" --include='*.conf' --include='*/' --exclude='*' \
    "${MINERVA_ROOT}/var/nginx/" "${DEST}/"
  echo "copied: nginx from ${MINERVA_ROOT}/var/nginx${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 3g. Fluentd
DEST="${BACKUP_ROOT}/fluentd"
SVC=$(docker service ls --format '{{.Replicas}}|{{.Name}}' 2>/dev/null | awk -F'|' '$1 !~ /(^0\/|\/0$)/ {print $2}' | grep -E '(^|_)fluentd$' | head -1)
SVC_NOTE="${SVC:+ (service=${SVC})}"; SVC_NOTE="${SVC_NOTE:- (service not running)}"
if [ -z "$(find "${MINERVA_ROOT}/var/fluentd" -maxdepth 1 -type f -name '*.conf' -print -quit 2>/dev/null)" ]; then
  echo "skip: no fluentd drop-in *.conf files present" | tee -a "${BACKUP_ROOT}/probe.out"
else
  rsync -a "${RSYNC_EXCLUDES[@]}" --include='*.conf' --exclude='*' \
    "${MINERVA_ROOT}/var/fluentd/" "${DEST}/"
  echo "copied: fluentd from ${MINERVA_ROOT}/var/fluentd${SVC_NOTE}" | tee -a "${BACKUP_ROOT}/probe.out"
fi

# 4. Manifest
find "${BACKUP_ROOT}" -type f -print | sort > "${BACKUP_ROOT}/manifest.files"
if command -v sha256sum >/dev/null; then
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 sha256sum > "${BACKUP_ROOT}/manifest.sha256"
else
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 shasum -a 256 > "${BACKUP_ROOT}/manifest.sha256"
fi

# 5. Archive
cd "$(dirname "${BACKUP_ROOT}")"
tar czf "third-party-config-${TS}.tar.gz" "$(basename "${BACKUP_ROOT}")"
ls -lh "third-party-config-${TS}.tar.gz"

# 6. Summary
ARCHIVE="$(dirname ${BACKUP_ROOT})/third-party-config-${TS}.tar.gz"
echo
echo "================ Third-Party Config Backup Summary ================"
echo "Timestamp:    ${TS}"
echo "Host:         $(hostname -f)"
echo "Backup dir:   ${BACKUP_ROOT}"
echo "Archive:      ${ARCHIVE}"
echo "Archive size: $(du -h "${ARCHIVE}" | cut -f1)"
echo "File count:   $(wc -l < "${BACKUP_ROOT}/manifest.files")"
echo
echo "Per-item result:"
sed 's/^/  /' "${BACKUP_ROOT}/probe.out"
echo "==================================================================="

Kubernetes 

# 1. Confirm context
kubectl config current-context
kubectl auth can-i get minervainfras.minerva.minerva.nvidia.com -A
kubectl auth can-i get minervaapps.minerva.minerva.nvidia.com -A

# 2. Prepare
export TS=$(date -u +%Y%m%dT%H%M%SZ)
export BACKUP_ROOT="${HOME%/}/manual-backups/third-party-config-k8s/${TS}"
mkdir -p "${BACKUP_ROOT}/custom-resources" "${BACKUP_ROOT}/context"
: > "${BACKUP_ROOT}/probe.out"

# 3. Context
{
  echo "=== date ===";          date -u
  echo "=== kubectl version ==="; kubectl version
  echo "=== cluster-info ===";  kubectl cluster-info
  echo "=== nodes ===";         kubectl get nodes -o wide
  echo "=== minervainfras ==="; kubectl get minervainfras -A
  echo "=== minervaapps ===";   kubectl get minervaapps -A
} > "${BACKUP_ROOT}/context/k8s-context.txt"

# 4. Export CRs
for KIND in minervainfras minervaapps; do
  if ! kubectl get crd "${KIND}.minerva.minerva.nvidia.com" >/dev/null 2>&1; then
    echo "skip: CRD ${KIND} not installed" | tee -a "${BACKUP_ROOT}/probe.out"
    continue
  fi
  if [ -z "$(kubectl get "${KIND}" -A -o name 2>/dev/null | head -1)" ]; then
    echo "skip: no ${KIND} resources found" | tee -a "${BACKUP_ROOT}/probe.out"
    continue
  fi
  kubectl get "${KIND}" -A -o yaml > "${BACKUP_ROOT}/custom-resources/${KIND}.yaml"
  echo "copied: ${KIND}" | tee -a "${BACKUP_ROOT}/probe.out"
done

# 5. Manifest + archive
find "${BACKUP_ROOT}" -type f -print | sort > "${BACKUP_ROOT}/manifest.files"
if command -v sha256sum >/dev/null; then
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 sha256sum > "${BACKUP_ROOT}/manifest.sha256"
else
  find "${BACKUP_ROOT}" -type f -not -name manifest.sha256 -print0 \
    | xargs -0 shasum -a 256 > "${BACKUP_ROOT}/manifest.sha256"
fi
cd "$(dirname "${BACKUP_ROOT}")"
tar czf "third-party-config-k8s-${TS}.tar.gz" "$(basename "${BACKUP_ROOT}")"
ls -lh "third-party-config-k8s-${TS}.tar.gz"

# 6. Summary
ARCHIVE="$(dirname ${BACKUP_ROOT})/third-party-config-k8s-${TS}.tar.gz"
echo
echo "============ Third-Party Config Backup (Kubernetes) Summary ============"
echo "Timestamp:    ${TS}"
echo "Context:      $(kubectl config current-context)"
echo "Backup dir:   ${BACKUP_ROOT}"
echo "Archive:      ${ARCHIVE}"
echo "Archive size: $(du -h "${ARCHIVE}" | cut -f1)"
echo "File count:   $(wc -l < "${BACKUP_ROOT}/manifest.files")"
echo
echo "Per-item result:"
sed 's/^/  /' "${BACKUP_ROOT}/probe.out"
echo "========================================================================"

Last updated: