Spectrum-X NCCL Plugin

Overview

The Spectrum-X NCCL Plugin provides a set of plugins designed to optimize NVIDIA’s NCCL library for Spectrum-X and Quantum platforms. It enables more resilient and higher-performance communication across these platforms. In HPC-X, the plugin is located at: $HPCX_DIR/nccl_spectrum-x_plugin/lib. 

Loading the Plugin

For NCCL to detect the network plugin, add the plugin path to the library search path environment variable.

 The plugin can be loaded by explicitly setting the library search path using LD_LIBRARY_PATH: 

$ export LD_LIBRARY_PATH=$HPCX_DIR/nccl_spectrum-x_plugin/lib:$LD_LIBRARY_PATH$ <run command>

With HPCX, the plugin can also be loaded by NCCL's environment variable NCCL_NET_PLUGIN=spcx

Features

• Network Failure Recovery  - Automatically detects and recovers from communication link failures, ensuring uninterrupted distributed training and system stability without user intervention.

• Dynamic Network Load Balance -  Dynamically redistributes data across multiple network interfaces based on available throughput and communication patterns, preventing bottlenecks and improving overall transfer efficiency.

• Topology Awareness – Automatically detects long-haul connections using network topology information and applies optimized transport settings to enhance overall network performance.

• NCCL Profiler Plugin – Enables detailed monitoring of NCCL’s internal network activity during communication.

Resiliency and Load Balancing

The Spectrum-X NCCL Plugin provides enhanced resiliency and load-balancing capabilities when multiple network devices (ports or NICs) are available per GPU in AI workloads. Starting from version 1.2, the plugin also considers the communication pattern being executed by NCCL, like number of peers and traffic per peer, along with device load characteristics to provide optimal distribution of traffic across multiple network devices.  

Topology Injection and Awareness

Starting with version 1.2, the Spectrum-X NCCL Plugin introduces a mechanism to specify network topology and transport parameters. The plugin supports specifying topology through a file format similar to Slurm’s topology format, where each entry defines connections between switches and hosts with relative latency or cable-length values.

Based on this topology, the plugin automatically identifies long-haul connections and takes advantage of NVIDIA XGS configuration for these. It also improves the NCCL algorithm selection model by taking the latencies across the long-haul connections into account.

For more information, refer to NVIDIA XGS.

Configurable Bandwidth Loss Limits

Starting with version 1.3, the Spectrum-X NCCL Plugin introduces configurable thresholds for acceptable bandwidth degradation during transparent failover. Applications can specify multi-tier limits where higher bandwidth loss triggers shorter tolerance windows before a fatal error is raised. Configuration is done through the NCCL_IB_NIC_BW_LOSS_LIMITS environment variable, which accepts comma-separated pairs of loss percentage and duration in minutes:

$ export NCCL_IB_NIC_BW_LOSS_LIMITS=25:2,50:1

This example configuration triggers a fatal error if bandwidth loss reaches 25% for 2+ minutes, and 50% for 1+ minute, allowing brief performance drops during recovery while preventing long-running jobs from continuing with severely degraded performance.

SHARP

Plugin supports Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) - an in-network computing technology on InfiniBand Quantum switches. 

The following environment variable enables the SHARP aggregation with NCCL when using the plugin.

NCCL_COLLNET_ENABLE=1

The following environment variable enables SHARP allgather overlap when using the plugin. This is useful when SHARP‑based Reduce‑Scatter is enabled, so the Reduce‑Scatter ↔ Allgather phase can overlap.

SHARP_COLLNET_OVERLAP_AG=1

Note: In this release, the bundled NCCL tuner enables CollNetDirect AllGather only at PPN=1; for multi-PPN runs, set NCCL_TUNER_PLUGIN=none before NCCL initialization.

GPU-Initiated Networking (GIN)

The Spectrum-X (SPCX) NCCL Plugin adds GIN support on top of the standard Spectrum-X network plugin. GIN allows GPU kernels to initiate network put/signal operations. Load the SPCX plugin as described in before.

GIN backends

With NCCL_GIN_ENABLE=1, set NCCL_GIN_TYPE to select the backend exposed by the Spectrum-X plugin:

• Proxy — NCCL_GIN_TYPE=2. Host-side progress over InfiniBand verbs. 

• GDAKI — NCCL_GIN_TYPE=3. GPU-initiated path using DOCA GPUNetIO. 

• GPI — NCCL_GIN_TYPE=4. General-push interface uses Spectrum-X software and hardware capabilities, and in this release, it is based on an experimental GPI interface.

GIN GPI Requirements (Experimental GPI Interface)

• GPI is supported only through the Spectrum-X plugin; set NCCL_NET_PLUGIN=spcx to enable it. GPI requires NCCL 2.30.7 or later, a Spectrum-X switch and a Connect-X NIC (CX8+) with GPUDirect RDMA enabled (peermem or DMA-BUF).

• For X86-based system, GPI requires setting the PeerMappingOverride=1

  • $ cat /etc/modprobe.d/nvidia.conf
    options nvidia NVreg_RegistryDwords="PeerMappingOverride=1;"

• The current GPI backend uses an experimental GPI interface intended for early enablement and is subject to change.

• Current GPI release is supported only with physical-function (PF) NICs. Virtual-function (VF) and VF+PF hybrid configurations are not supported.

• For MPI/UCX launches, if UCX and SPCX GPI use the same HCA ports, UCX device-memory allocations can prevent GPI channel creation. Set `UCX_IB_DM_COUNT=0` before MPI initialization.

Default GIN backend

If NCCL_GIN_TYPE is unset, the Spectrum-X plugin uses auto-selection:

• Try GDAKI (GPUNetIO) first when the plugin is built with GDAKI support, GPUNetIO-capable devices are present, and GPU Direct RDMA requirements are met. Log line: GIN: using GDAKI backend.

• If GDAKI cannot be used (no devices or no GDR), fall back to Proxy (host verbs). Log line: GIN: using Proxy backend.

• GPI is not auto-selected. Set NCCL_GIN_TYPE=4 explicitly to use GPI.

 
Environment variables — Spectrum-X / NCCL

Core settings for GIN with the Spectrum-X plugin (assuming SPCX plugin has already been selected by NCCL):

All other NCCL GIN parameters

Connection counts, context counts, GPUNetIO library path, symmetric GIN kernels, and other tuning (for example NCCL_GIN_NCONNECTIONS, NCCL_GIN_NCONTEXTS, NCCL_GIN_GPUNETIO_PATH, NCCL_SYM_GIN_KERNELS_ENABLE) are defined in the NVIDIA NCCL User Guide. Use the NCCL documentation for those variables.

 Please visit NCCL Documentation

Known limitations

The following apply to the current Spectrum-X plugin and HPC-X drop. They are targeted for fix in the next HPC-X release.

• GPI + symmetric collectives + CTA limit. Symmetric NCCL collectives over GIN (NCCL_SYM_GIN_KERNELS_ENABLE=1) on GPI require NCCL_SYM_CTAS of 32 or less. Higher CTA counts are not supported on the current GPI path and can cause failures on symmetric ReduceScatter / AllGather. Until the next HPC-X release, keep NCCL_SYM_CTAS<=32 for GPI symmetric workloads, or avoid symmetric GIN collectives on GPI except for validation.

• GPI may run into a memory allocation issue due to limited resources. If you see something like Cannot allocate memory error, try reducing the channel size (ginQueueDepth) and/or the number of gin contexts (ginContextCount) in the ncclDevCommRequirements structure

• GPI on InfiniBand. If GPI fails during QP connection with an INIT→RTR error, set this temporary workaround before launch: GWQ_USE_GRH=1

NetInspector Profiler

The Spectrum-X NCCL Plugin extends the NCCL Inspector profiler with NetInspector, which adds network-level instrumentation and telemetry.

NetInspector provides:

• Collective performance – Per-collective metrics including algorithm bandwidth, bus bandwidth, message sizes (full logical and per-rank), execution time, and timing source (GPU/CPU). Supports optional verbose mode with detailed event sequence numbers and timestamps per kernel channel. With DTS export, collective metrics are also accumulated and exported as counters (nccl_coll_*).

• Transaction-level metrics – Per-transaction send/receive data including chunk sizes, resent bytes, div sizes (for app-aware load balancing), peer addressing, and collective/P2P context. Requires NCCL_INSPECTOR_TELEMETRY_LEVEL=2.

• Burst bandwidth telemetry – Real-time bandwidth measurements per InfiniBand device using exponential moving averages (EMA). Export cadence is controlled by NCCL_INSPECTOR_BURST_BANDWIDTH_FLUSH_INTERVAL_MS.

• Burst slowdown detection – Configurable threshold-based alerts when burst bandwidth falls below expected levels, with deficit tracking (deficit_gbps).

• Queue Pair and Load Balancing telemetry – QP create events and cumulative load-balancing (CLB) weight metrics per device. Weights are updated on each QP weight change and also heartbeated on the dynamic-flow resample interval. Load-balance weight-asymmetry variance between planes is also exported.

• Dynamic-Flow telemetry – Periodic resample of 12 InfiniBand failure/recovery counters (nccl_ib_*_total) that track sender/receiver recovery lifecycle, backup device activation/deactivation, persistent bandwidth-loss failures, and QPs reaching minimum allowed bandwidth.

• Device resilience – Device failure and recovery events with total bandwidth, active device count, and bandwidth delta tracking.

• Link error monitoring – Link error events with IB work completion status codes and cumulative link-error counts per device.

• Async thread events – InfiniBand asynchronous event monitoring (port state changes, errors, etc.) captured from the IB async thread.

Export Formats

NetInspector supports two export backends:

JSON Lines (.jsonl) — NCCL_INSPECTOR_DUMP_TYPE=json

Offline, per-event records with microsecond-level timestamps. Output directory defaults to nccl-inspector-unknown-jobid or nccl-inspector-<SLURM_JOBID> under SLURM (overridable via NCCL_INSPECTOR_DUMP_DIR).

DTE (DOCA Telemetry Exporter) — NCCL_INSPECTOR_DUMP_TYPE=dts

Prometheus-compatible metrics via CollectX. Discrete device/link/burst/async events require optional OTLP logs (NCCL_INSPECTOR_DTS_OTLP_LOGS=1).

Metrics can be exported by pull (Prometheus scrape endpoint) or push (Prometheus Remote Write / PRW), or both:

Prometheus scrape endpoint (pull) — CollectX exposes a local HTTP metrics endpoint for Prometheus (or another scraper) to pull from:

export NCCL_INSPECTOR_DUMP_TYPE=dts
export PROMETHEUS_ENDPOINT="http://0.0.0.0:9091"

Then scrape http://<host>:9091/metrics from Prometheus.

Prometheus Remote Write / PRW (push) — CollectX pushes time-series samples to a remote Prometheus-compatible receiver:

export NCCL_INSPECTOR_DUMP_TYPE=dts
export CLX_REMOTE_WRITE_RECEIVER="http://RECEIVER_HOST:RECEIVER_PORT/api/v1/write"

# For exporting via Prometheus Remote Write protocol v2, add:
export CLX_REMOTE_WRITE_CONTENT_TYPE="application/x-protobuf;proto=io.prometheus.write.v2.Request"

# Recommended for short jobs so shutdown flush can complete:
export CLX_REMOTE_WRITE_METRICS_SHUTDOWN_FLUSH_TIMEOUT_MS=500

# For exporting OTLP logs to an open telemetry receiver
export NCCL_INSPECTOR_DTS_OTLP_LOGS=1
export CLX_OPEN_TELEMETRY_RECEIVER="http://RECEIVER_HOST:RECEIVER_PORT/"

Environment Variables

Metrics (DTS / Prometheus)

Logs (DTS OTLP; requires NCCL_INSPECTOR_DTS_OTLP_LOGS=1)

Last updated: September 01, 2026