Spectrum-X NCCL Plugin

Overview

The Spectrum-X NCCL Plugin provides a set of plugins designed to optimize NVIDIA’s NCCL library for Spectrum-X and Quantum platforms. It enables more resilient and higher-performance communication across these platforms. In HPC-X, the plugin is located at: $HPCX_DIR/nccl_spectrum-x_plugin/lib.

Loading the Plugin

For NCCL to detect the network plugin, add the plugin path to the library search path environment variable.

 The plugin can be loaded by explicitly setting the library search path using LD_LIBRARY_PATH: 

$ export LD_LIBRARY_PATH=$HPCX_DIR/nccl_spectrum-x_plugin/lib:$LD_LIBRARY_PATH$ <run command>

With HPCX, the plugin can also be loaded by NCCL's environment variable NCCL_NET_PLUGIN=spcx

Features

• Network Failure Recovery  - Automatically detects and recovers from communication link failures, ensuring uninterrupted distributed training and system stability without user intervention.

• Dynamic Network Load Balance -  Dynamically redistributes data across multiple network interfaces based on available throughput and communication patterns, preventing bottlenecks and improving overall transfer efficiency.

• Topology Awareness – Automatically detects long-haul connections using network topology information and applies optimized transport settings to enhance overall network performance.

• NCCL Profiler Plugin – Enables detailed monitoring of NCCL’s internal network activity during communication.

Resiliency and Load Balancing

The Spectrum-X NCCL Plugin provides enhanced resiliency and load-balancing capabilities when multiple network devices (ports or NICs) are available per GPU in AI workloads. Starting from version 1.2, the plugin also considers the communication pattern being executed by NCCL, like number of peers and traffic per peer, along with device load characteristics to provide optimal distribution of traffic across multiple network devices.  

Topology Injection and Awareness

Starting with version 1.2, the Spectrum-X NCCL Plugin introduces a mechanism to specify network topology and transport parameters. The plugin supports specifying topology through a file format similar to Slurm’s topology format, where each entry defines connections between switches and hosts with relative latency or cable-length values.

Based on this topology, the plugin automatically identifies long-haul connections and take advantage of NVIDIA XGS configuration for these. It also improves the NCCL algorithm selection model taking the latencies across the long-haul connections into account.

For more information, refer to NVIDIA XGS.

NetInspector Profiler

The Spectrum-X NCCL Plugin extends the NCCL Inspector profiler with NetInspector, which adds network-level instrumentation and telemetry.

See plc/items/id_9e97f808/profiler.md in the repository for the most up-to-date information on the NetInspector Profiler.

NetInspector provides:

• Transaction-level metrics – Per-transaction send/receive data including chunk sizes, resent bytes, div sizes (for app-aware load balancing), and peer addressing.

• Burst bandwidth telemetry – Real-time bandwidth measurements per InfiniBand device using exponential moving averages (EMA with configurable beta).
  

• Burst slowdown detection – Configurable threshold-based alerts when burst bandwidth falls below expected levels, with deficit tracking.
  

• Queue Pair (QP) lifecycle – QP creation, weight updates, and cumulative load-balancing weight metrics per device.
  

• Device resilience – Device failure and recovery events with total bandwidth, active device count, and bandwidth delta tracking.
  

• Link error monitoring – Link error events with IB work completion status codes and cumulative link-error counts per device.
  

• Async thread events – InfiniBand asynchronous event monitoring (port state changes, errors, etc.) captured from the IB async thread.
  

• Collective performance – Per-collective metrics including algorithm bandwidth, bus bandwidth, message sizes, execution times, and timing source (GPU/CPU). Supports optional verbose mode with detailed event sequence numbers and timestamps per kernel channel.

Export Formats

NetInspector supports two export backends:

• JSON Lines (.jsonl) – Microsecond-level timestamps in JSON Lines format for performance analysis

• DTS (DOCA Telemetry Service) – OTLP-compatible metrics and logs export for integration with observability platforms (Prometheus, Grafana, etc.)

Environment Variables

Last updated: June 23, 2026