Networking NVIDIA UFM Enterprise User Manual

Known Issues in This Release

Internal Ref.

Issue

5384856

Description: After upgrading to UFM Enterprise 6.27.0, the time from UFM start to the end of the initial fabric discovery (grid_start) increases on very large fabrics. In simulated-fabric tests, the time increased by about 26% on an 8.4K-node fabric (from 218 seconds to 275 seconds) and by about 79% on a 7.5K-node fabric (from 28 seconds to 50 seconds). The first start after installation is the slowest. Later starts are faster but remain longer than in UFM Enterprise 6.26.

Workaround: N/A

Keywords: Performance, grid start, startup time, discovery, large fabric, regression, UFM 6.27

Detected in version: 6.27.1

5366704

Description: In derived Multi-NIC host mode, moving a card from one host to another existing host by changing its node description can give the destination host the host ID of the source host. The destination host loses its own host ID, and the remaining card of the source host appears as a new host. Events 348 and 349 report a rename instead of a move. Per-host data (custom name, IP, credentials, groups) may be lost. The outcome depends on the node counts and the GUID order.

Workaround: Restore the original node descriptions to get the original host IDs back. Avoid moving a card to another existing host by renaming it.

Keywords: Multi-NIC, derived host, host ID, HCA moved, event 348, event 349, node description

Detected in version: 6.27.1

5415248

Description: In UTM mode, PEMI SNR collection by secondary telemetry may trigger firmware failures on affected XDR switches. This issue applies to standalone UFM deployments on bare metal or Docker, without UFM Infra. UTM on Kubernetes with XDR is not supported in this release.

Workaround: Temporarily disable PEMI SNR collection as follows:

  1. For a Docker deployment, first open a shell inside the UFM container:

docker exec -it ufm /bin/bash
  1. Back up the secondary counterset, remove the PEMI SNR counters, and restart secondary telemetry:

Bash
utm_xcset=/opt/ufm/gvvm/telemetry/low_freq_debug.xcset

cp -pn "$utm_xcset" \
  /opt/ufm/files/conf/secondary_telemetry_defaults/low_freq_debug.xcset.before-pemi-snr

sed -i -E \
  '/^[[:space:]]*snr_(media|host)(_db)?_lane[0-7][[:space:]]*=/d' \
  "$utm_xcset"

/etc/init.d/ufmd ufm_telemetry_secondary_restart

UFM automatically uploads the modified counterset, and UTM recreates the secondary telemetry instances using it. No separate upload is required.

  1. Inside the UTM plugin container, check the /data/<collection>/conf/launch_ibdiagnet_config.ini file of each recreated secondary collection. It must contain:

plugin_env_CLX_EXPORT_API_SKIP_PEMI_SNR_SAMPLES=1

Neither arg_pemi_snr_samples nor arg_ibd_pemi_snr_samples should be present.

Note: The restart temporarily interrupts secondary telemetry collection. Media and host SNR counters are unavailable while the workaround is applied. If the affected UTM version is still in use, reapply the workaround after a UFM upgrade or UFM container recreation.

Keywords: UTM, Secondary Telemetry, PEMI SNR, XDR

Detected in version: 6.27.1

Last updated: