Networking NVIDIA HPC-X Software Toolkit

Running, Configuring and Rebuilding HPC-X

The sources for SHMEM and OMPI can be found at $HPCX_HOME/sources/ .

Please refer to $HPCX_HOME/sources/ and HPC-X README file for more information on building details.

Rebuilding Open MPI

Rebuilding Open MPI Using a Helper Script 

The $HPCX_HOME/utils/hpcx_rebuild.sh script can rebuild OMPI and UCX from HPC-X using the same sources and configuration. It also takes into account HPC-X's environments: vanilla, MT and CUDA.

For details, run:

$HPCX_HOME/utils/hpcx_rebuild.sh --help

Rebuilding Open MPI from HPC-X Sources

HPC-X package contains Open MPI sources that can be found in $HPCX_HOME/sources/ folder. Further information can be found in HPC-X README file.

Procedure_Heading_Icon.PNG To build Open MPI from sources:

$ HPCX_HOME=/path/to/extracted/hpcx
$ source $HPCX_HOME/hpcx-init.sh
$ hpcx_load
$ cd $HPCX_HOME/sources/
$ rm -rf openmpi-gitclone
$ tar -zxf openmpi5-gitclone.tar.gz
$ cd openmpi-gitclone
$ ./configure --prefix=${HPCX_HOME}/hpcx-ompi \
            --with-hcoll=${HPCX_HOME}/hcoll \
            --with-ucx=${HPCX_HOME}/ucx \
            --with-ucc=${HPCX_HOME}/ucc \
            --with-platform=contrib/platform/mellanox/optimized \
            --with-slurm
$ make -j9 all && make -j9 install


Open MPI and OpenSHMEM are pre-compiled with UCX and HCOLL, and use them by default.

If HPC-X is intended to be used with SLURM PMIx plugin, Open MPI should be built against external PMIx, Libevent and HWLOC and the same Libevent and PMIx libraries should be used for both SLURM and Open MPI.

Additional configuration options:

--with-pmix=<path-to-pmix>
--with-libevent=<path-to-libevent>
--with-hwloc=<path-to-hwloc>

Running MPI with HCOLL

HCOLL is disabled by default in HPC-X.

  • Running with default HCOLL configuration parameters:

    $ mpirun -mca coll_hcoll_enable 1 -x HCOLL_MAIN_IB=mlx5_0:1 <...>
    
  • Running OSHMEM with HCOLL:

    % oshrun -mca scoll_mpi_enable 1 -mca scoll basic,mpi -mca coll_hcoll_enable 1 <...>
    
    

Direct Launch of Open MPI and OpenSHMEM using SLURM 'srun'

The default HPC-X is not built with SLURM support. In order to use a direct launch with srun, rebuild the OpenMPI or HPC-X with the slurm version installed on the system:

  • Open MPI:

    `env <MPI/OSHMEM-application-env> srun --mpi={pmi2|pmix} <srun-args> <mpi-app-args>`
    


All Open MPI/OpenSHMEM parameters that are supported by the mpirun/oshrun command line can be provided through environment variables using the following rule:

"-mca <param_name> <param-val>" => "export OMPI_MCA_<param_name>=<param-val>"

For example an alternative to "-mca coll_hcoll_enable 1" with 'mpirun' is
"export OMPI_MCA_coll_hcoll_enable=1" with 'srun '


 Process and Memory Affinity with Open MPI v5.0

Starting with Open MPI v5.0, the runtime environment transitioned from ORTE to the PMIx Reference RunTime Environment (PRRTE). Processor and memory affinity management is now handled by PRRTE, which operates as an independent submodule.

While mapping and binding still utilize hwloc and the familiar --map-by <object> and --bind-to <object> directives, the default mapping policy has changed from map-by-socket to map-by-core.

Processor affinity must be enabled for memory affinity to function. An unbound process may migrate off its local NUMA memory and lose locality performance benefits.

Default Mapping and Binding Policy: OMPI v4.1.x vs. v5.0.x

Feature / Behavior

OMPI v4.1.x (ORTE)

OMPI v5.0.x (PRRTE 3.x)

Default Mapping

--map-by numa (falls back to socket, then slot)

--map-by core

Default Binding

--bind-to numa

--bind-to core

Placement (N ≪ cores)

Ranks round-robin across NUMA domains—load is spread across sockets.

Ranks placed sequentially on consecutive cores—N ranks all land on socket 0 until that socket fills.

Practical Impact

For multi-threaded MPI codes, the v5.0.x default packs all ranks on the first socket. As a result, threads of every rank will contend for the exact same memory subsystem. Conversely, the v4.1.x default spreads one rank per NUMA domain, leaving each rank a full NUMA's worth of cores for its respective threads.

Recovering v4.1.x-Equivalent Behavior Under v5.0.x

To recover the legacy behavior, pass the mapping and binding rules explicitly.

For NUMA-granularity placement:

mpirun --map-by numa --bind-to numa -n N <app>

For socket-granularity placement:

mpirun --map-by socket --bind-to socket -n N <app>

Passing only --bind-to core does not restore spread placement. When a binding is specified without an explicit mapping, v5.0.x automatically sets the mapping to match the binding object (--bind-to core ⇒ --map-by core), producing the same packed layout on socket 0.

Always pair it with an explicit --map-by ... flag if you need a specific rank distribution across your hardware topology.

References

Last updated: