UFM Telemetry Manager (UTM) Plugin

Plugin Release Notes

Changes and New Features

Bug Fixes

Overview

The UFM Telemetry Manager (UTM) plugin partitions InfiniBand fabric monitoring across multiple UFM Telemetry Instances (TIs) for high-scale clusters. UTM assigns fabric ports to TIs deterministically using consistent hashing, optionally with redundancy, and manages their health and port assignments as the fabric changes.

Key capabilities:

• Stable port distribution: each port is assigned to a specific TI by consistent hashing, so the port-to-TI mapping does not reshuffle on every TI restart.

• Configurable redundancy: a port can be monitored by multiple TIs simultaneously (port_redundancy_factor), so a TI failure causes zero monitoring gap on its ports.

• TI failure handling: failed TIs are kept in the active assignment during a grace period to absorb transient failures; if the TI does not recover, its ports are redistributed across the surviving TIs.

• Port labels: telemetry samples can be enriched with labels generated from the fabric topology and with operator-defined labels pushed through the UTM REST API.

UTM runs two telemetry groups by default: primary (high-frequency port counters) and secondary (broader counter set, lower frequency). Each group independently covers 100% of the fabric. UFM controls how many instances are created in each through the [Telemetry] section of gv.cfg; see UFM Clustered Telemetry.

Deployment

UTM is deployed as a UFM plugin. Two deployment paths are supported.

UFM Plugin Mode

The UTM plugin can be added either via the Command Line Interface or the Web UI.

CLI Deployment

To add the plugin:

/opt/ufm/scripts/manage_ufm_plugins.sh add -p utm

To remove the plugin:

/opt/ufm/scripts/manage_ufm_plugins.sh remove -p utm

Web-UI Deployment

1. Navigate to the UFM Web UI and click Settings in the left panel.

2. Open the Plugin Management tab.

3. Right-click on the UTM plugin row and select Add.

4. Open Telemetry Status in the left panel to access the UTM UI.

To stop the plugin: in Plugin Management, right-click the UTM row and select Disable.

Kubernetes Deployment

For deploying UTM in Kubernetes alongside UFM Enterprise, see UFM Clustered Telemetry.

Configuration

The UTM configuration file utm_config.ini is located at /opt/ufm/files/conf/plugins/utm/utm_config.ini. UTM restarts its main process automatically when the file changes.

Key Tunables

The settings most operators tune. Anything not listed here ships with a sensible default and should not normally need to change.

Authentication

UTM authenticates to the UFM REST API using either token or username/password. Token is preferred where available; username/password is the fallback.

Token authentication (recommended):

Write the API token to a file and point ufm_token_file at it:

[ufm]
ufm_token_file = /config/ufm_token

If the file exists and is non-empty, UTM uses token auth automatically.

Username/password authentication (fallback):

For non-default UFM credentials:

[ufm]
ufm_user = <user>
ufm_pass = <password>

UTM falls back to username/password when no token file is configured.

Port Labels

UTM can add labels to the telemetry collected for each fabric port. These labels become additional dimensions in the exported telemetry and can be used to filter, group, and correlate port metrics.

UTM supports two independently configurable label sources:

• Topology labels are derived from the InfiniBand fabric topology. When topology labels are enabled, the UTM master generates one shared label set at startup and whenever the topology changes.

• Custom labels are supplied by an operator or an external automation system through the UTM REST API. Examples include rack, tenant, owner, workload, or maintenance-domain labels.

When both sources are enabled, UTM serves a combined labels payload to every TI in the target telemetry group. TIs refresh the payload periodically (about every 30 seconds); updating custom-label values does not require pushing the data separately to every TI, and a value change takes effect on the next refresh. If both sources define the same label key for a port, the custom value overrides the topology-derived value.

Enabling Port Labels

The following example enables topology labels and custom labels:

[general]
labels_enabled=true
labels_custom_enabled=true

These two settings are independent. Enable only labels_enabled for topology labels, only labels_custom_enabled for custom labels, or both for the combined payload.

UTM automatically selects the remaining behavior:

• Topology generation uses the packaged metadata generator.

• Non-HA deployments keep custom labels in memory and require a re-push after the state is lost.

• HA deployments persist custom labels automatically on the existing UFM shared file system so both nodes serve the same data.

• Request size, store size, refresh interval, and the loopback metadata endpoint use product-defined limits and defaults rather than configuration settings.

Port 8899 is reserved for the loopback-only metadata endpoint used by co-located TIs and must not be assigned to a TI or another local service.

Changing either label switch does not rewrite the environment of an already-running TI. After the configuration change, check the custom-label status endpoint and recreate the affected TIs during an appropriate maintenance window. The configuration change is complete when migration_pending is false.

Custom-Label Format

Send custom labels as pipe-delimited text, with one fabric port per line:

<node-guid>_<port-number>|<label-key>|<label-value>|<label-key>|<label-value>

For example:

248a070300abcdef_1|rack|rack-12|host|compute-07|owner|platform
248a070300abcdef_2|rack|rack-12|host|compute-08|owner|ai-training

The <node-guid>_<port-number> identifier must exactly match the identifier the topology label file uses for that port (bare lowercase hexadecimal GUID, no 0x prefix). When a port's identifier matches, UTM merges its custom labels into that port's topology labels: on a shared label key the custom value wins, and the port's other topology labels are preserved. A custom identifier that does not match any topology port is served as its own standalone set.

Label keys and values cannot contain the pipe (|) delimiter, embedded newlines, or any other control character (tabs, NUL, and similar bytes are rejected). Blank lines and lines beginning with # are ignored, so a submitted file may contain comments. A request is rejected if it is malformed, exceeds 8 MiB, or would grow the store beyond 20,000 (port, key) entries. The 8 MiB and 20,000-entry limits are totals across all telemetry groups, and the 8 MiB limit bounds both a single request and the entire stored set. These are fixed product safety limits.

Consuming Labeled Telemetry

Labels are exposed on each TI's telemetry endpoints on demand, through the /labels/ URL prefix. This mirrors UFM's legacy telemetry behavior: the default counter stream a TI serves — for example /csv/metrics, or the /csv/cset/<name> set that UFM's own data plane scrapes — stays label-free, and a consumer that wants labels requests the /labels/ variant of the same path. UTM sets no additional exporter flags, so enabling port labels never changes the shape of the counters UFM already collects.

• /labels/metrics returns the counters in Prometheus text format with every label — topology and custom — attached as native label dimensions. This is the recommended surface for Prometheus/Grafana consumers, and the only surface that carries operator-defined custom label keys that are not part of the topology label set.

• /labels/csv/metrics returns the same counters in CSV form with the labels added as columns. The CSV column set is fixed to the topology label keys, so a custom label appears here when it overrides the value of an existing topology key. A custom label with a new key name is served in the payload and appears on /labels/metrics, but does not become a new CSV column.

Managing Custom Labels

The examples below use UFM plugin mode. Replace default with the telemetry group to update. Group names are path components and must be URL-encoded when they contain reserved characters.

Replace all custom labels for a group:

curl -k -u <user>:<pass> \
  -X POST \
  --data-binary @custom-labels.txt \
  'https://<UFM_HOST>/ufmRest/plugin/utm/host/labels/default?mode=replace'

Merge the submitted labels into the existing custom labels:

curl -k -u <user>:<pass> \
  -X POST \
  --data-binary @custom-labels.txt \
  'https://<UFM_HOST>/ufmRest/plugin/utm/host/labels/default?mode=merge'

In merge mode, submitted (port, key) pairs replace the current values for those pairs; unmentioned pairs remain unchanged. An empty replace request clears the group, while an empty merge request makes no changes.

Retrieve the exact merged payload currently served for a group:

curl -k -u <user>:<pass> \
  'https://<UFM_HOST>/ufmRest/plugin/utm/host/labels/default'

This authenticated operator endpoint returns the same topology-plus-custom payload that TIs retrieve through their node-local listener. It is read-only and is available on both HA nodes.

Custom-label mutation (POST and DELETE) response codes:

Mutation requests must send an explicit Content-Length (--data-binary @file does this). Chunked or streamed request bodies are rejected with 411, so a real push is never coerced into an empty replace that would clear the group.

Check custom-label state and propagation:

curl -k -u <user>:<pass> \
  'https://<UFM_HOST>/ufmRest/plugin/utm/host/labels/status/default'

The status endpoint remains available when custom labels are disabled so that an operator can monitor TIs that still require recreation. Mutation requests are rejected while labels_custom_enabled=false.

The status response contains the following fields:

Clear the custom labels for a group:

curl -k -u <user>:<pass> \
  -X DELETE \
  'https://<UFM_HOST>/ufmRest/plugin/utm/host/labels/default'

Restart and HA Behavior

• In non-HA mode, custom labels are held in memory. After a UTM process or pod restart, the status endpoint reports initialized=false, and the operator or automation must push the labels again. Topology labels continue to be served while custom labels are absent.

• In HA mode, UTM automatically stores custom labels atomically on the existing shared file system. Writes are accepted only by the current master, but both HA nodes serve the persisted labels to their local TIs. A failover does not require a re-push. If the shared location is unavailable or not writable, UTM disables custom labels and continues with topology labels only.

• As a fail-safe, if port 8899 cannot be bound while custom labels are being enabled, UTM disables custom labels and keeps TIs on topology labels. If custom labels were just disabled and an existing TI still points to the old loopback URL, UTM reports a degraded migration state and that TI must be recreated immediately. Check the UTM status and log if custom labels appear inactive after a restart.

• Re-push only when initialized=false (or when a tracked store_epoch indicates that the store was replaced). Do not re-push solely because loaded=false or process_epoch changed; doing so could restore labels that were intentionally cleared.

• Deployment mapping. Durability follows the HA switch, not the platform. A bare-metal HA appliance (active-active) persists custom labels on the shared file system and survives failover with no re-push. Every non-HA UTM (enable_ha=0) keeps custom labels in memory and must be re-pushed after a restart — this covers both the default Kubernetes clustered-telemetry deployment (a single plugin UTM for the full fabric) and standalone UTMs each pinned to one fabric segment. Push, and re-push, custom labels to each UTM that owns the ports being labeled.

The port used by TIs to retrieve labels is a loopback-only implementation endpoint. It is not exposed as an operator API and does not replace the authenticated UFM plugin REST path shown above.

GUI

The Telemetry Status page is accessible from the UFM Web UI sidebar under Telemetry Status. In standalone mode, it is also available directly at https://<utm-host>:8888/files/index.html using UTM's generated certificate.

The page contains:

• Top pane: general info; controls to add a TI URL for monitoring; refresh-interval selector.

• Group panes: one panel per telemetry group, showing every TI in the group with status and counters.

• Bottom pane: system events with history navigation.

TI Status Fields

TI Management Actions

Right-click a TI row to:

• Pause: pause a running TI; its ports are redistributed to other TIs in the group.

• Resume: resume a paused TI.

• Exclude: pause and remove the TI from its group (the TI itself stays on the host). Empty groups are removed automatically.

REST API

All GUI features (TI management, monitoring, configuration) are accessible via REST.

Accessing the API

In UFM plugin mode (proxied through UFM):

curl -k -u <user>:<pass> https://<UFM_HOST>/ufmRest/plugin/utm/<COMMAND>

Direct (e.g. K8s pod, port-forward):

curl -k https://<UTM_HOST>:8888/<COMMAND>

In plugin mode UTM listens on plain HTTP on port 8888; HTTPS termination is handled by UFM's proxy. Standalone UTM exposes HTTPS with a generated certificate.

Common Commands

The examples below use the standalone HTTPS form; substitute the proxied form for plugin mode.

# List all UTM endpoints
curl -k https://127.0.0.1:8888/help

# Status of monitored TIs
curl -k https://127.0.0.1:8888/status

# Add an externally-running TI to a monitoring group
curl -k 'https://127.0.0.1:8888/add_server?url=http://127.0.0.1:9001&group=primary'

# Pause / resume / remove a monitored TI
curl -k 'https://127.0.0.1:8888/pause_server?url=http://127.0.0.1:9001'
curl -k 'https://127.0.0.1:8888/start_server?url=http://127.0.0.1:9001'
curl -k 'https://127.0.0.1:8888/remove_server?url=http://127.0.0.1:9001'

# Spawn TIs by count, with automatic round-robin HCA allocation
curl -k -X POST 'https://127.0.0.1:8888/host/create_sessions?group=primary&count=2&sample_rate=30'

# Stop a TI by session id
curl -k 'https://127.0.0.1:8888/host/remove_telemetry?session_id=<id>'

POST /host/create_sessions response codes: 200 at least one session created, 400 invalid params, 409 group already has running instances, 500 all failed, 503 no HCA in Active+LinkUp state.

Port 8899 is reserved for the loopback metadata endpoint (see Port Labels) and cannot be assigned to a TI. A create_telemetry request for that port is rejected with 409, and a create_sessions port range that lands on it skips that one instance.

Last updated: August 27, 2026