UFM on Kubernetes

• UFM Enterprise Kubernetes Deployment Guide
  • Overview
    • What's New
  • Supported Environments
    • Kubernetes Version
    • Node Operating Systems
    • Hardware Requirements
  • Prerequisites
    • Kubernetes Cluster
    • Helm
    • Storage
    • InfiniBand
    • Network Ports
  • Installation
    • Step 1: Set Up Storage
    • Step 2: UFM Docker Image
    • Step 3: Create Namespace and License ConfigMap
    • Step 4: Install UFM with Helm
    • Step 5: Verify Installation
    • Configuration Reference
    • Namespace Configuration
    • Image Configuration
    • UFM Configuration
    • Storage Configuration
    • Resource Limits (Required)
    • License Configuration
    • Startup Probe Configuration
    • Liveness Probe Configuration
    • Service Configuration
    • Ingress Configuration
    • Scheduling Configuration
    • Plugin Configuration
  • Deployment Options
    • Option 1: Host Network Mode (Default)
    • Option 2: With Ingress Controller
      • Step 1: Install Ingress Controller (if not installed)
      • Step 2: Deploy UFM with Ingress
    • Option 3: Using a Values File
  • Plugin Deployment
    • Plugin Configuration Fields
    • Deploy Single Plugin
    • Deploy Multiple Plugins
    • Plugin Without HTTP Port
    • Plugins with Values File
  • Custom Configuration Files
    • Included Config Files
    • Method 1: Edit Files Before Install
      • Extract the chart:
      • Edit config files:
      • Install with modified files
    • Configuration Priority
    • Adding Custom Counter Sets
      • Extract the chart:
      • Add custom cset file:
      • Install:
  • Operations
    • Start/Stop UFM
      • Stop UFM
      • Start UFM
    • View Logs
      • Container Logs
      • UFM Application Logs
    • Access UFM UI and REST API
      • Web UI
      • REST API
  • Uninstallation
    • Remove UFM
    • Resource Cleanup
      • Remove All Resources
    • Remove Specific Resources Only
  • Networking
    • Port Reference
    • Host Network Architecture
    • Traffic Flow with Ingress
  • Storage
    • Data Persistence
  • Security Considerations
    • Privileged Container Requirement
    • Host Network Implications
  • Best Practices
  • Monitoring
    • Kubernetes Probes
    • Health Check Details
    • Monitoring Commands
      • Verify Probe Status
      • Verify UFM Processes
      • Check UFM Health Log
  • Known Limitations

UFM Enterprise Kubernetes Deployment Guide

This guide provides comprehensive instructions for deploying NVIDIA UFM Enterprise on Kubernetes using Helm charts.

Overview

What's New

UFM Enterprise now supports deployment on Kubernetes clusters using Helm charts. This deployment method provides:

• Declarative Configuration: Define your UFM deployment using Helm values

• Simplified Operations: Use standard Kubernetes tools for deployment, upgrades, and management

• Plugin Support: Deploy UFM plugins as separate pods with automatic configuration

• Ingress Integration: Expose UFM through Kubernetes Ingress controllers

• Persistent Storage: Use Kubernetes PersistentVolumeClaims for data persistence

Supported Environments

Kubernetes Version

Kubernetes 1.28 or later.

Node Operating Systems

UFM on Kubernetes supports the same operating systems as UFM Enterprise. See the Installation Notes for the complete list of supported operating systems.

Hardware Requirements

UFM on Kubernetes has the same hardware requirements as UFM Enterprise. See the Installation Notes for detailed specifications.

Prerequisites

Before deploying UFM Enterprise on Kubernetes, ensure the following requirements are met:

Kubernetes Cluster

• Kubernetes cluster version 1.28 or later

• kubectl configured with cluster access

• Cluster admin permissions for installation

Helm

• Helm 3.x installed on the management workstation. Run: 

  # Verify Helm installation
  helm version

 Storage

• A StorageClass that supports ReadWriteMany access mode

• Minimum 10GB storage capacity

InfiniBand

• At least one node with InfiniBand interface

• DOCA drivers installed on the worker node where UFM will be deployed

• InfiniBand port configured and in "up" state. Run: 

  # Verify InfiniBand interface
  ip link show | grep -E 'ib[0-9]|ibp'

UFM License

• Valid UFM Enterprise license file

• License file accessible from the management workstation

Network Ports

UFM uses host network mode. Ensure these ports are available on the target node (more ports might be used):

Installation

Step 1: Set Up Storage

UFM requires ReadWriteMany storage. Make sure you have a Persistent storage configured.

Step 2: UFM Docker Image

The UFM Docker image needs to be located in a place your K8S cluster has access to.

It can be pre loaded or in your own Registry.

Step 3: Create Namespace and License ConfigMap 

# Create the namespace
kubectl create namespace ufm-enterprise

# Create license ConfigMap
kubectl create configmap ufm-license \
  --from-file=<license-filename>.lic=/path/to/your/<license-filename>.lic \
  -n ufm-enterprise

Step 4: Install UFM with Helm 

helm install ufm ufm-enterprise-<version>-helm.tgz \
  --namespace ufm-enterprise \
  --set config.fabricInterface=<your_ib_interface> \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4

Replace <your_ib_interface> with your InfiniBand interface name (e.g., ib0, ibp4s0f0).

Step 5: Verify Installation

Watch the pod status: 

kubectl get pods -n ufm-enterprise -w

Expected state transitions:

NAME                              READY   STATUS        AGE
ufm-ufm-enterprise-xxxxxxxxxx     0/1     Init:0/1      5s
ufm-ufm-enterprise-xxxxxxxxxx     0/1     PodInitializing   30s
ufm-ufm-enterprise-xxxxxxxxxx     0/1     Running       45s
ufm-ufm-enterprise-xxxxxxxxxx     1/1     Running       2m

 Configuration Reference

All configuration options are set via Helm values. Use --set key=value or a values file (-f values.yaml).

Namespace Configuration

Image Configuration

UFM Configuration

Storage Configuration

Resource Limits (Required)

License Configuration

Startup Probe Configuration

Liveness Probe Configuration

Service Configuration

Ingress Configuration

Scheduling Configuration

Example - Schedule on specific node: 

--set nodeSelector."kubernetes\.io/hostname"=ufm-node

Plugin Configuration

Example - Deploy with a plugin: 

helm install ufm ufm-enterprise-<version>-helm.tgz \
  --namespace ufm-enterprise \
  --set config.fabricInterface=ib0 \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4 \
  --set plugins.items[0].name=<plugin-name> \
  --set plugins.items[0].image=<plugin-image> \
  --set plugins.items[0].tag=<plugin-version> \
  --set plugins.items[0].port=<plugin-port> \
  --set plugins.items[0].imagePullPolicy=Always

Deployment Options

Option 1: Host Network Mode (Default)

This is the default and simplest deployment mode. UFM binds directly to the host's network ports. 

helm install ufm ./ufm-enterprise \
  --namespace ufm-enterprise \
  --set config.fabricInterface=ib0 \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4

Access UFM at: https://<node-ip>:443

Option 2: With Ingress Controller

Use an Ingress controller for external access with TLS termination and hostname-based routing.

Step 1: Install Ingress Controller (if not installed)

Step 2: Deploy UFM with Ingress 

helm install ufm ./ufm-enterprise \
  --namespace ufm-enterprise \
  --set config.fabricInterface=ib0 \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4 \
  --set ingress.enabled=true \
  --set ingress.className=traefik \
  --set ingress.host=ufm.example.com

Access UFM at: https://ufm.example.com

Option 3: Using a Values File

For complex configurations, use a YAML values file: 

# my-values.yaml
namespace:
  name: ufm-enterprise

image:
  pullPolicy: Never

config:
  fabricInterface: ib0
  mgmtInterface: eth0

storage:
  className: nfs-client
  size: 50Gi

resources:
  requests:
    memory: 8Gi
    cpu: 4
  limits:
    memory: 16Gi
    cpu: 8

license:
  existingConfigMap: ufm-license

ingress:
  enabled: true
  className: nginx
  host: ufm.example.com
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

nodeSelector:
  kubernetes.io/hostname: ufm-node

Deploy with the values file: 

helm install ufm ./ufm-enterprise -f my-values.yaml -n ufm-enterprise

Plugin Deployment

UFM plugins run as separate pods with pod affinity to ensure they are scheduled on the same node as UFM.

Plugin Configuration Fields

Deploy Single Plugin

helm install ufm ./ufm-enterprise \
  --namespace ufm-enterprise \
  --set config.fabricInterface=ib0 \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4 \
  --set plugins.items[0].name=<plugin-name> \
  --set plugins.items[0].image=<plugin-image> \
  --set plugins.items[0].tag=<plugin-version> \
  --set plugins.items[0].port=<plugin-port> \
  --set plugins.items[0].imagePullPolicy=Always

Deploy Multiple Plugins 

helm install ufm ufm-enterprise-<version>-helm.tgz \
  --namespace ufm-enterprise \
  --set config.fabricInterface=ib0 \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set license.existingConfigMap=ufm-license \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4 \
  --set plugins.items[0].name=<plugin1-name> \
  --set plugins.items[0].image=<plugin1-image> \
  --set plugins.items[0].tag=<plugin1-version> \
  --set plugins.items[0].port=<plugin1-port> \
  --set plugins.items[0].imagePullPolicy=Always \
  --set plugins.items[1].name=<plugin2-name> \
  --set plugins.items[1].image=<plugin2-image> \
  --set plugins.items[1].tag=<plugin2-version> \
  --set plugins.items[1].port=<plugin2-port> \
  --set plugins.items[1].imagePullPolicy=Always

Plugin Without HTTP Port

Some plugins don't expose an HTTP port. Omit the port field: 

--set plugins.items[0].name=<plugin-name> \
--set plugins.items[0].image=<plugin-image> \
--set plugins.items[0].tag=<plugin-version> \
--set plugins.items[0].imagePullPolicy=Always

Plugins with Values File

# plugins-values.yaml
plugins:
  items:
    - name: <plugin-name>
      image: <plugin-image>
      tag: <plugin-version>
      port: <plugin-port>
      imagePullPolicy: Always

helm install ufm ufm-enterprise-<version>-helm.tgz -f my-values.yaml -f plugins-values.yaml -n ufm-enterprise

Custom Configuration Files

The Helm chart includes default UFM configuration files that can be customized.

Included Config Files

Method 1: Edit Files Before Install

Extract the chart:

tar xzf ufm-enterprise-<version>-helm.tgz

Edit config files:

vim ufm-enterprise/files/config/gv.cfg
vim ufm-enterprise/files/config/opensm/opensm.conf

Install with modified files

helm install ufm ./ufm-enterprise -n ufm-enterprise \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4

Configuration Priority

Configuration is applied in this order (later wins):

1. Base install/upgrade - UFM default config files

2. Helm chart config files - Files from files/config/ directory

3. Helm values - config.fabricInterface, config.mgmtInterface

Adding Custom Counter Sets

Add custom Prometheus counter set files for telemetry customization:

Extract the chart:

tar xzf ufm-enterprise-<version>-helm.tgz

Add custom cset file:

mkdir -p ufm-enterprise/files/config/telemetry_defaults/prometheus_configs/cset/
cp my-custom-counters.cset ufm-enterprise/files/config/telemetry_defaults/prometheus_configs/cset/

Install:

helm install ufm ./ufm-enterprise -n ufm-enterprise \
  --set storage.className=nfs-client \
  --set image.pullPolicy=Never \
  --set resources.requests.memory=4Gi \
  --set resources.requests.cpu=2 \
  --set resources.limits.memory=8Gi \
  --set resources.limits.cpu=4

Operations

Start/Stop UFM

Stop UFM

Scale down the deployment to 0 replicas: 

kubectl scale deployment -n ufm-enterprise -l app=ufm-enterprise --replicas=0

Verify UFM is stopped:

kubectl get pods -n ufm-enterprise

Start UFM

Scale back up to 1 replica:

kubectl scale deployment -n ufm-enterprise -l app=ufm-enterprise --replicas=1

Wait for the pod to be ready:

kubectl get pods -n ufm-enterprise -w

View Logs

Container Logs

Follow logs:

kubectl logs -n ufm-enterprise -l app=ufm-enterprise -f

Previous container logs (after crash):

kubectl logs -n ufm-enterprise -l app=ufm-enterprise --previous

UFM Application Logs

Access UFM UI and REST API

Web UI

REST API 

# Get UFM version
curl https://<node-ip>:443/ufmRest/app/ufm_version

# List resources
curl https://<node-ip>:443/ufmRest/resources/systems

Uninstallation

Remove UFM

Run: 

helm uninstall ufm -n ufm-enterprise

Resource Cleanup

Remove All Resources

To delete the entire namespace and all associated resources: 

kubectl delete namespace ufm-enterprise

Remove Specific Resources Only

To delete selected resources instead of the full namespace:

kubectl delete pvc -n ufm-enterprise -l app.kubernetes.io/name=ufm-enterprise
kubectl delete configmap -n ufm-enterprise ufm-license
kubectl delete secret -n ufm-enterprise ufm-tls

Networking

Port Reference

Host Network Architecture

UFM is deployed with hostNetwork: true, enabling direct access to:

• InfiniBand interfaces for fabric management

• Host ports for external connectivity

• Low-latency communication with OpenSM

Implications:

• UFM pods bind directly to the node’s network stack

• Required ports must be available on the host

• Port conflicts may prevent pod startup

Traffic Flow with Ingress

Client → Ingress Controller (80/443) → UFM Service → UFM Pod (8080/18443)

When Ingress is enabled:

• UFM listens internally on ports 8080 and 18443

• The Ingress controller handles external traffic on ports 80 and 443

• TLS may be terminated at the Ingress or passed through to UFM

Storage

Data Persistence

All data stored under /opt/ufm/files/ is persisted via a PersistentVolumeClaim (PVC), ensuring data retention across pod restarts.

Security Considerations

Privileged Container Requirement

UFM runs in privileged mode to allow:

• Direct access to InfiniBand hardware

• Loading of kernel modules

• Management of the InfiniBand Subnet Manager

Host Network Implications

Using hostNetwork: true means:

• UFM can access all host network interfaces

• Service ports are exposed directly on the node

• Kubernetes NetworkPolicies do not apply to pod traffic

Best Practices

1. Dedicated Nodes – Deploy UFM on dedicated infrastructure nodes.

2. Node Taints – Apply taints to prevent unrelated workloads from scheduling on UFM nodes.

3. Network Segmentation – Isolate UFM nodes on a management network.

4. RBAC Controls – Restrict access to the UFM namespace using Kubernetes RBAC.

5. Secrets Management – Store sensitive data in Kubernetes Secrets.

6. Regular Updates – Keep UFM and Kubernetes components up to date.

Monitoring

Kubernetes Probes

UFM uses two probes:

Health Check Details

Startup Probe:

• Calls /app/versioning/ on UFM Web server

• Returns 503 during initialization, 200 when ready

• Allows up to 5 minutes for startup

Liveness Probe:

• Verifies UfmHealthRunner process is running

• Checks for failover flag (critical failure indicator)

• Verifies config_watcher.sh is running

Monitoring Commands

Verify Probe Status

Run the following command to review the liveness and startup probe configuration and status: 

kubectl describe pod -n ufm-enterprise -l app=ufm-enterprise | grep -A 5 -E "Liveness:|Startup:"

Verify UFM Processes

Use the command below to list running UFM-related processes inside the pod:

kubectl exec -n ufm-enterprise -l app=ufm-enterprise -- ps aux

Check UFM Health Log

Run the following command to inspect the UFM health log file: 

kubectl exec -n ufm-enterprise -l app=ufm-enterprise -- cat /opt/ufm/files/log/ufmhealth.log

Known Limitations

Last updated: March 12, 2026