HPC-X Toolkit Change Log History
|
Category |
Change |
|---|---|
|
Rev 2.25 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release:
|
|
Spectrum-X NCCL Plugin |
Added support for optimized NCCL communication on Spectrum-X and Quantum platforms, providing enhanced resiliency, dynamic load balancing, topology awareness, and advanced profiling capabilities. |
|
Bug Fixes |
See Bug Fixes section. |
|
Rev 2.24.1 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release:
|
|
Bug Fixes |
See Bug Fixes. |
|
Rev 2.23 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release:
|
|
NCCLNet Plugin |
Added support for NCCLNet plugin integration to enhance NCCL performance and resilience on Spectrum-X and Quantum platforms. For further information, see Spectrum-X NCCL Plugin section. |
|
Rev 2.22.1 |
|
|
ConnectX-8 and XDR Multi-Plane Networks Support |
Introducing hardware-level support for Direct Data Placement (DDP), which maximizes the use of all network lanes to achieve the full bandwidth capacity of XDR networks. |
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release:
|
|
Rev 2.22.0 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Rev 2.21.1 |
|
|
Support for Direct Data Placement in mlx5dv and DevX Interfaces |
Added support for direct data placement (DDP) across both mlx5dv and DevX interfaces. This enhancement enables out-of-order receives and achieves full wire speed on ConnectX-8 adapters. |
|
Adapter Cards |
Added support for ConnectX-8 SuperNIC. |
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Rev 2.21.0 |
|
|
Hybrid Mode for Hardware DCS Offload |
Added support for hardware DCS offload in hybrid mode, which combines existing software DCS with hardware DCS offload. To enable this feature, set |
|
GGA Transport for High-Speed Data Transfer |
Added support for GGA transport, a high-speed DMA copy engine that enables efficient data transfer between host memory and the DPU's internal memory. |
|
Multi-Node NVLink with Optimized Protocol Selection |
Added support for multi-node NVLink, with automatic detection and selection of the most efficient data transfer protocols. |
|
IP Address Filtering for RoCE Devices |
Added support for filtering RoCE devices based on their IP address, allowing the selection of specific network subnets. To configure the filter, set |
|
Automatic Selection of GPU Bounce Buffers for Large Message Transfers |
Added support for an optimization that automatically selects GPU bounce buffers for large message transfers when these buffers offer performance benefits over host memory buffers. |
|
Single Memory Key Creation Using ODP for Enhanced Efficiency |
Added support for a feature that leverages the ODP capability to create a single memory key for the entire process's virtual address space. This reduces the number of allocated memory keys, helping to bypass firmware limitations. To enable this feature, set |
|
MLNX_OFED to DOCA-OFED Transition |
Starting this version, the host driver is part of the NVIDIA DOCA package. DOCA-OFED is a DOCA-Host profile that includes the same components, drivers, and tools as MLNX_OFED. Installing DOCA-OFED will result with the same file system on the host as MOFED. For further information, please see NVIDIA MLNX_OFED to DOCA-OFED Transition Guide. |
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Rev 2.20.0 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
ClusterKit |
Revised the ClusterKit section to include updated details and added subsections. |
|
Rev 2.19.0 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
UCC Grace Support and Performance Profile |
Added performance improvements of broadcast and reduce collectives on NVIDIA Grace CPU by optimizing parameters of shared memory transport (TL/SHM). |
|
Unified Collective Communication (UCC) |
Added the CPU/GPU Bcast feature which implements the MCAST Bcast algorithm in UCC. Refer to Unified Collective Communication (UCC). |
|
Unified Communication - X Framework Library |
Added the on-demand paging feature. Refer to On-demand Paging (ODP) |
|
ClusterKit |
Added the CPU/GPU stress testing feature. Refer to ClusterKit |
|
Bug Fixes |
See Bug Fixes. |
|
Rev 2.18.0 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
HPC-X and Singularity |
Deprecated support for HPC-X singularity containerization. |
|
OSU Micro-Benchmarks Path |
Removed version from the OSU Micro-Benchmarks path. The current one is ompi/tests/osu-micro-benchmarks and ompi/tests/osu-micro-benchmarks-cuda |
|
Known Issues |
See Known Issues. |
|
Rev 2.17.1 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Known Issues |
See Known Issues. |
|
Rev 2.17.0 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
Added the following Supported Platforms and OSs:
|
|
Supported Cards |
Added support for GH100. |
|
Known Issues |
See Known Issues. |
|
Rev 2.16.2 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Supported Cards |
All cards up to BlueField-3 and ConnectX-7. |
|
Bug Fixes |
|
|
Rev 2.16 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Supported Cards |
Added support for BlueField-3 cards. |
|
Bug Fixes |
See Bug Fixes. |
|
Rev 2.15 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
Bug Fixes |
See Bug Fixes. |
|
Rev 2.14 |
|
|
TL/UCP Special Service Worker |
Added support for having a separate UCX UCP worker use UCC service collectives. For further information, please see TL/UCP Special Service Worker section. |
|
Data Type Support in CUDA Executor Component (EC) |
Added out-of-box support for all datatypes and reduction operations for UCC collectives for GPUs. For further information, please see Data Type Support in CUDA Executor Component section. |
|
EC/CUDA One-shot Kernel with Cooperative Launch |
Added support for using a single CUDA kernel for CUDA operations in UCC GPU collectives. For further information, please see EC/CUDA One-shot Kernel with Cooperative Launch section. |
|
Out-Of-Box Native GPU Allreduce |
Added support for the UCC library to detect the NVIDIA NVLink topology and select the best GPU-based algorithms for supported collectives (Allgather/v, Reducescatter/v). For further information, please see Out-Of-Box Native GPU Allreduce section. |
|
Bug Fixes |
See Bug Fixes. |
|
Rev 2.13.1 LTS |
|
|
Operating System |
Added support for Ubuntu v20.04 and v20.10. |
|
Rev 2.13 |
|
|
HPC-X Content |
Updated HPC-X Content section to reflect the communication libraries versions embedded in this HPC-X release.
|
|
NCCL-RDMA-SHARP-PLUGIN |
Added support for NCCL plugin API v5. |
|
SHARP |
Added support for SHARP on NDR. |
|
Bug Fixes |
See Bug Fixes section. |
|
Rev 2.12 |
|
|
UCX |
Added a method to set RoCE ECE value from UCX configuration. For example: UCX_IB_ECE=auto will use maximal ECE value, and UCX_IB_ECE= will use a specific numeric ECE value. |
|
HPC-X Content |
Updated the version of the UCX communication library to v1.14. |
|
Rev 2.11 |
|
|
Adapter Cards |
Added support NVIDIA ConnectX-7 adapter card with with 400 Gb/s speed. |
|
SHARPD |
sharpd daemon process has been removed. sharpd-related activity is now performed from the user application process |
|
HPC-X Content
|
Updated the versions of the following communication libraries.
|
|
Added support for UCC, a collective communication operations API and library in HPC-X. UCC is now part of the HPC-X package. For further information on UCC, pleased see Unified Collective Communication (UCC) section. |
|
|
Rev 2.10 |
|
|
UCX |
Added support for atomics on GPU memory target |
|
OpenSHMEM |
Added support for reducing memory overhead on scale |
|
Rev 2.9 |
|
|
UCX Configuration File |
The UCX configuration file enables the user to apply configuration variables set by the user in the /etc/ucx/ucx.conf file. For further information see UCX Configuration File. |
|
Instrumentation and Monitoring FUSE-based Tool |
This new functionality enables the user to analyze UCX-based applications in runtime. The tool is based on Filesystem in Userspace (FUSE) interface. If the feature is enabled, a directory for each process using UCX will be created in /tmp/ucx. For further information see Instrumentation and Monitoring FUSE-based Tool. |
|
OS Architecture |
HPC-X v1.9 onwards will no longer support PPC architecture in its releases. |
|
Bug Fixes |
|
|
Rev 2.8 |
|
|
HPC-X Content |
Updated the following communication libraries and acceleration packages versions:
|
|
UCX
|
Added support for Multi-interface for cloud (client-server) applications. |
|
Added support for using Adaptive-Routing (out-of-order) on an SL that supports it. |
|
|
Added support for UCP Active-Messages API with Rendezvous. |
|
|
Added support for Keepalive functionality on the UCT layer. |
|
|
Performed several error handling enhancements. |
|
|
Added support for GPU-NIC locality discovery. |
|
|
NCCL-RDMA-SHARP-PLUGIN
|
Added support for NCCL Plugin API v4. |
|
Added support for PCIe Relaxed Ordering. |
|
|
Added support for Adaptive Routing. |
|
|
Rev 2.7 |
|
|
UCX
|
Added a new request API. For further information on this request API, please refer to UCX API documentation. |
|
Added support for PCIe Relaxed Ordering. |
|
|
Added out-of-box support for RoCE LAG. |
|
|
Added Flow Control support for RDMA Read operations. |
|
|
AMD Rome optimizations: Optimized IB connection establishment procedures to reduce system noise. |
|
|
Rev 2.6 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions:
|
|
UCX
|
Added support in UCX for communication between containers configured to share the memory namespaces. |
|
Added strided Receive queue support for hardware tag matching. |
|
|
Made the following performance improvements on AMD EPYC servers.
|
|
|
Added support for multithreaded memory region in Open SHMEM (OSHMEM) applications to improve performance in job startup and teardown latencies. The multithreaded MR enables a more efficient use of the CPU resource during registration of memory regions larger than 4GB.
|
|
|
Cuda |
Removed Cuda support in SLES 11 and RHEL 6 OSs. |
|
Rev 2.5 |
|
|
HPC-X Content
|
Updated the following communications libraries and acceleration packages versions:
|
|
Removed CUDA init script (hpcx-init-cuda.sh), and environmental module (modules/hpcx-cuda) from HPC-X. Up until HPC-X v2.4, these files used to point to the default files hpcx-init.sh and modules/hpcx. Now, these CUDA files no longer exist, and users can only use the default init script and environmental module for enabling CUDA support. |
|
|
CUDA |
Unified Vanilla and CUDA environments. CUDA v10.0 is supported out of the box with standard init script or environmental module. Note: HPC-X is compiled against CUDA version 10.0, which does not support GCC versions newer than v8. Therefore, HPC-X built on systems with GCC versions above v8 will not have CUDA support. |
|
UCX
|
Made performance optimizations. |
|
Added full support for rdma-core. |
|
|
Added support for CUDA v10.1. |
|
|
Rev 2.4 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions:
|
|
Removed |
|
|
CUDA |
Unified Vanilla and CUDA environments. CUDA v9 is supported out of the box with standard init script or environmental module. Note: HPC-X is compiled against CUDA version 9, which does not support GCC versions newer than v7. Therefore, HPC-X built on systems with GCC versions above v7 will not have CUDA support. |
|
UCX |
Enabled HDR, SocketDirect and MultiRail features out-of-box. |
|
UCX Random DCI is now at GA level. |
|
|
Implemented a number of job startup optimizations. |
|
|
Added support from PCIe atomic operations feature. |
|
|
HCOLL |
Added support for performing floating point 16 bit operations for machine learning scenarios. |
|
OpenMPI |
Added multi threading support to OpenMPI OSC UCX. |
|
General |
HPC-X is now available through the EasyBuild framework: https://easybuild.readthedocs.io/en/latest/ |
|
Rev 2.3 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions:
|
|
UCX
|
UCX is now compiled without JAVA bindings. |
|
Added support for running UCX over rdma-core, for DC transport and direct verbs. |
|
|
Emulation layer: Added the ability to run UCX over software emulation of remote memory access and atomic operations. This provides full support of SHMEM and MPI-RMA over shared memory, TCP, and older RDMA hardware, such as ConnectX-3 HCA. |
|
|
HCOLL
|
HCOLL and NVIDIA SHARP are now compiled with CUDA support. |
|
Added support for CUDA buffers over SRA allreduce algorithm. |
|
|
MXM |
Removed support for MXM library. |
|
OpenMPI
|
Added the following configuration options to OMPI:
|
|
Updated the configuration file platform/mellanox/optimized config in OMPI upstream by removing BTL OpenIB and UCT support and removing links to MXM/FCA usage. |
|
|
Removed PMI2 support. |
|
|
Rev 2.2 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions: • NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) version 1.7 • HCOLL version 4.1 • UCX version 1.4 |
|
Added support for Singularity containerization.
|
|
|
“osc ucx” is no longer the default one-sided-component in OpenMPI. |
|
|
Removed KNEM library from HPC-X package. UCX will use the KNEM available in MLNX_OFED. |
|
|
MXM Support |
Open MPI and HCOLL are not compiled with MXM anymore. Both are compiled with UCX only and use it by default. |
|
UCX |
Added support for the following UCX features: • New API for establishing client-server connection. • Out-of-box support for Memory In Chip (MEMIC) on ConnectX-5 HCAs. |
|
HPC-X Setup |
Added support for HPC-X to work on Huawei ARM architecture. |
|
HCOLL |
Improved performance by utilizing zero-copy messaging for MPI Bcast. |
|
Rev 2.1 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions: • Open MPI version 3.1.x • NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) version 1.5 • HCOLL version 4.0 • MXM version 3.7 • UCX version 1.3 • OpenSHMEM v1.3 specification compliant |
|
UCX |
• UCX is now the default pml layer for Open MPI, default spml layer for OpenSHMEM, and default OSC component for MPI RMA. • Added the following UCX features: • Added support for GPU memory in UCX communication libraries • Added support for Multi-Rail protocol |
|
MXM |
The UD_RNDV_ZCOPY parameter is set to ‘no’ by default. This means that the zcopy mechanism for the UD transport is disabled when using the Rendezvous protocol. |
|
HCOLL |
• UCX is now the default p2p transport in HCOLL • Improved multi-threaded performance • Improved shared memory performance • Added support for NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) v1.5 • Added support for NVIDIA SHARP software multi-channel/multi-rail capable algorithms • Improved Allreduce large message algorithm • Improved AlltoAll algorithm |
|
Profiling IB verbs API (ibprof) |
Removed ibprof tool from HPC-X toolkit. |
|
UPC |
Removed UPC from HPC-X toolkit. |
|
Rev 2.0 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions: • OpenMPI version 3.0.0 • Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) version 1.4 • HCOLL version 3.9 • UCX version 1.3 |
|
UCX |
• UCX is now at GA level. • Added the following UCX features: • [ConnectX-5 only] Added support for hardware Tag Matching with DC transport. • [ConnectX-5 only] Added support for Out-of-order RDMA RC and DC to support adaptive routing with true RDMA. • Added UCX datatypes - community approved datatype support. • Added UCX support to Inbox RHEL. • Added GPUDirect RDMA support. • Hardware Tag Matching (See section Hardware Tag Matching in the User Manual) • SR-IOV Support (See section SR-IOV Support in the User Manual) • Adaptive Routing (AR) (See section Adaptive Routing in the User Manual) • Error Handling (See section Error Handling in the User Manual) |
|
HCOLL |
• Added support for Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) v1.4 • Added support for NCCL on-host GPU based collectives. • Added support for Hierarchical GPU based allreduce using NCCL for scale-in and MXM/UCX for scale-out. • Improved shared memory performance for allreduce, barrier, and broadcast. Targeting high thread count systems, e.g. Power9. • Improved large message allreduce (multi-radix, zero-copy fragmentation, CPU vectorization.) • Added new and improved AlltoAllv algorithm - hybrid logarithmic pair-wise exchange. • Added support for on-demand HCOLL memory. Improves HCOLL's memory footprint on high thread count system e.g. Power9. • Added a high performance multithreaded implementation to support MPI_THREAD_MULTIPLE applications. Designed specifically for high thread count systems, e.g. Power9. • HCOLL startup improvements. |
|
Open MPI / OpenSHMEM |
• Added support for Open MPI 3.0.0. • Added support for xpmem kernel module. • Added a high performance implementation of shmem_ptr() with UCX SPML. • Added a UCX allocator. The UCX allocator optimizes intra-node communication by allowing direct access to memories of processes on the same node. The UCX allocator can only be used with the UCX SPML. • Added a UCX one-sided component to support MPI RMA operations. |
|
Rev 1.9.7 |
|
|
Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) |
Bug Fixes, see Section 4, “Bug Fixes History”, on page 11 |
|
Rev 1.9 |
|
|
HPC-X Content |
Updated the following communications libraries and acceleration packages versions: • OpenMPI version 2.1.2a1 • Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) version 1.3.1 • HCOLL version 3.8.1652 • MXM version 3.6.3103 • UCX version 1.2.2947 |
|
UCX |
Point-to-point communication API, with tag matching, remote memory access, and atomic operations. This can be used to implement MPI, PGAS, and Big Data libraries and applications- IB transport |
|
A cleaner API with lower software overhead which provides better performance especially for small messages. |
|
|
Support for multitude of InifiniBand transports and NVIDIA offloads to optimize data transfer performance: • RDMA • DC • Out-of-order • HW tag matching offload • Registration cache • ODP |
|
|
Shared memory communications for optimal intra-node data transfer: • SysV • posix • knem • CMA • xpmem |
|
|
MXM |
Enabled Adaptive Routing for all the transport layers (UD/RC/DC). |
|
Memory registration optimization. |
|
|
Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) |
Improved the Out-of-the-box performance of Scalable Hierarchical Aggregation and Reduction Protocol (SHARP). |
|
Shared memory |
Improved the intranode performance of allreduce and barrier. |
|
Configuration |
Changed many default parameter setting in order to achieve best out-of-the-box experience for several applications including - CP2K, miniDFT, VASP, DL-POLY, Amber, Fluent, GAMES-UK, and LS-DYNA. |
|
FCA |
As of HPC-X v1.9, FCA v2.5 is no longer included in the HPC-X package. |
|
Improved AlltoAllv algorithm. |
|
|
Improved large data allreduce. |
|
|
Improved UCX BCOL. |
|
|
OS architecture |
Added support for ARM architecture. |
FCA Change Log History
|
Category |
Change |
|---|---|
|
Rev 3.5 |
|
|
FCA Collective |
Added MPI Allgatherv and MPI reduce |
|
FCA |
Added support for NVIDIA SHARP library (including SHARP allreduce, reduce and barrier) |
|
Enhanced scalability for CORE-Direct based collectives |
|
|
Added support for complex data types |
|
|
Rev 3.4 |
|
|
General |
UCX support |
|
Communicator caching scheme with eviction: improves jobstart and communicator creation time |
|
|
Collectives |
Collectives: Added Alltoallv and Alltoall small message algorithms. |
|
Rev 3.3 |
|
|
General |
Ported to PowerPC |
|
Thread safety added |
|
|
Collectives |
Improved large message allreduce algorithm (Enabled by default) |
|
Beta version of network topology awareness (Enabled by default) |
|
|
Rev 3.0 |
|
|
Collectives |
Offload collectives communication from MPI process onto NVIDIA interconnect hardware. |
|
Efficient collectives communication flow optimized to job and topology |
|
|
MPI collectives |
Significantly reduce MPI collectives runtime |
|
MPI-3 |
Native support for MPI-3 |
|
Blocking and Non-blocking collectives |
Support for blocking and nonblocking collectives |
|
HCOLL |
Supports hierarchical communication algorithms (HCOLL) |
|
Collective algorithm |
Supports multiple optimizations within a single collective algorithm |
|
Performance |
Increase CPU availability and efficiency for increased application performance |
|
MPI libraries |
Seamless integration with MPI libraries and job schedulers |
|
Rev 2.5 |
|
|
Multicast Group |
Added MCG (Multicast Group) cleanup tool |
|
Performance |
Performance improvements |
|
Rev 2.2 |
|
|
Performance |
Performance improvements |
|
Dynamic offloading rules |
Enabled dynamic offloading rules configuration based on the data type and reduce operations |
|
Mixed MTU |
Added support for mixed MTU |
|
Rev 2.1.1 |
|
|
AMD/Interlagos CPUs |
Added support for AMD/Interlagos CPUs |
|
Rev 2.1 |
|
|
Core-Direct® |
Added support for Core-Direct® technology (enables offloading collective operations to the HCA.) |
|
Non-contiguous data layouts |
Added support for non-contiguous data layouts |
|
PGI compilers |
Added support for PGI compilers |
Last updated: