UFM SLURM Integration | NVIDIA UFM Enterprise User Manual

Simple Linux Utility for Resource Management (SLURM) is a job scheduler for Linux and Unix-like kernels.

By integrating SLURM with UFM, you can:

Assign partition keys (PKeys) to SLRUM nodes that are assigned for specific SLURM jobs.
Create SHARP reservations based on SLURM nodes assigned for specific SLURM jobs.

Prerequisites

UFM 6.9.0 (or newer)
Python 3.0 on SLURM controller
UFM-SLURM integration files (provided independently)

Automatic Installation

A script is provided to install the UFM-SLURM integration automatically.

Using the SLURM controller, extract the UFM-SLURM integration tar file:
```
tar -xf ufm_slurm_integration.tar.gz
```
Run the installation script using root privileges.
```
sudo ./install.sh
```

Manual Installation

To install the UFM-SLURM integration manually:

Extract the UFM-SLURM integration tar file:
```
tar -xf ufm_slurm_integration.tar.gz
```
Copy the UFM-SLURM integration files to the SLURM controller folder.
Change the permissions of the UFM-SLURM integration files to 755.
Modify the SLURM configuration file on the SLURM controller, /etc/slurm/slurm.conf, and add/modify the following two parameters:
```
PrologSlurmctld=/etc/slurm/ufm-prolog.sh
EpilogSlurmctld=/etc/slurm/ufm-epilog.sh
```

UFM SLURM Config File

The integration process uses a configuration file located at /etc/slurm/ufm_slurm.conf. This file is used to configure settings and attributes for UFM-SLURM integration.

Here are the contents:

Attribute Name	Description	Optionality
`ufm_server`	IP of the UFM server to connect to	Mandatory
https_port	HTTPS port of the UFM server to connect with.	Optional, default is 443
`auth_type`	Should be `token_auth`, or `basic_auth` If you select `basic_auth`, you need to set `ufm_server_user` and `ufm_server_pass` If you select `token_auth`, you need to set `token_auth`	Mandatory
`ufm_server_user`	Username of the UFM server used to connect to UFM, if you set `auth_type=basic_auth`	Mandatory, depends on the `auth_type`
`ufm_server_pass`	UFM server user password	Mandatory, depends on the `auth_type`
`token`	Generated token when you set uth_typea to `token_auth`	Mandatory, depends on the `auth_type`
`pkey_allocation`	Controls whether UFM SLURM Integration allocates a Pkey for SLURM jobs. When set to false: No PKey is allocated; the default management PKEY (0x7fff) is used. When set to true: A PKey is allocated based on the '`pkey_allocation_mode`' parameter setting.	Mandatory, default is True
`pkey_allocation_mode`	The PKey allocation mode is either static or dynamic. The default value is static. Static: uses a statically assigned PKey when creating a new PKEY. Dynamic: uses a dynamic assignment when creating a new PKey based on the `slurm-job-id`.	Static or Dynamic, default is static
`pkey`	Hexadecimal string between "0x0001" and "0x7ffe" exclusive. Applicable only when assigning a static PKey for all SLURM jobs. If the `pkey_allocation_mode` is set to static, and the PKey is not set, the default management PKey (0x7fff) will be used.	Optional, default is "0x7fff" (This is the default management PKey)
`ip_over_ib`	PKey is a member of a multicast group that uses IP over InfiniBand	Hidden param, default is True
`index0`	If true, the API will store the PKey at index 0 of the PKey table of the GUID	Hidden param, default is False
`log_file_name`	Name of integration logging file	Optional

Configuring UFM for NVIDIA SHARP Allocation

To configure UFM for NVIDIA SHARP allocation/deallocation, you must set sharp_enabled and enable_sharp_allocation to true in the gv.cfg file.

Generate token_auth

If you set auth_type=token_auth in UFM SLURM’s config file, you must generate a new token by logging into the UFM server and running the following curl command:

curl -H "X-Remote-User:admin" -XPOST http://127.0.0.1:8000/app/tokens

Then you must copy the generated token and paste it into the config file beside the token_auth parameter.

Prolog and Epilog

Starting with UFM Enterprise v6.26.1, the UFM-SLURM integration relies solely on PKey membership. The prolog and epilog scripts:

ufm-prolog.sh – the prolog script is executed when a job is submitted and before running the job itself. It creates the partition key (pkey) assignment and/or NVIDIA SHARP reservation and assigns the SLURM job hosts for them.
ufm-epilog.sh – the epilog script is executed when a job is complete. It removes the partition key (PKey) assignment and/or NVIDIA SHARP reservation and frees the associated SLURM job hosts.

UFM's 'auto_reservation_by_pkey' handles the SHArP reservation lifecycle automatically:

Adding a host to a PKey triggers the creation of a matching SHArP reservation,
Removing the last host from that PKey tears the reservation down.

The direct SHArP-reservation code path previously used by the plugin
has been removed. The 'UFM SHArP allocation APIs' are scheduled for deprecation in the July 2026 release.

Integration Files

The integration uses scripts and configuration files to work, which should be copied to SLURM controller /etc”/slurm. Here is a list of these files:

File Name	Description
`ufm-prolog.sh`	Bash file which executes jobs related to UFM after the SLURM job is completed
`ufm-epilog.sh`	Bash file which executes jobs related to UFM before the SLURM job is executed
`ufm_slurm.conf`	UFM-SLURM integration configuration file
`ufm_slurm_prolog.py`	Python script that adds the SLURM job hosts to the configured PKey when the prolog bash script runs.
`ufm_slurm_epilog.py`	Python script that removes the SLURM job hosts from the PKey when the epilog bash script runs.
`ufm_slurm_utils.py`	Utility Python file containing functions and utilities used by the integration process

Running UFM-SLURM Integration

Using the SLURM controller, execute the following commands to run your batch job:

$ sbatch -N4 slurm_demo.sh
Submitted batch job 1

N4 is the number of compute nodes used to run the jobs. slurm_demo.sh is the job batch file to be run.

The output and result are stored in the working directory slurm-{id}.out, where {id} is the ID of the submitted job.

In the above example, after executing the sbatch command, you can see that the submitted job ID is 1. Therefore, the output file would be stored in slurm-1.out.

Execute the following command to see the output:

$cat slurm-1.out

On the UFM side, a PKey is created in case the pkey_allocation parameter is set to true in the configuration file. The pkey_allocation_mode parameter specifies the PKey creation type, which can either be static or dynamic. The default value is static.

Static: Uses a statically assigned PKey when creating a new PKey (taken from the pkey parameter, or the default management PKey 0x7fff if unset).

Dynamic: Uses a dynamic assignment when creating a new PKey based on the slurm-job-id (wrapped into the range 0x0001–0x7ffe).

The corresponding SHArP AM reservation is created and released automatically by UFM's auto_reservation_by_pkey mechanism, based on PKey membership changes. The UFM-SLURM Integration itself only manages PKey membership.

After the SLURM job is completed, the UFM-SLURM Integration removes the job's compute nodes from the PKey, and UFM automatically tears down the associated SHArP reservation once the PKey becomes empty.

From the moment a job is submitted by the SLURM server until its completion, a log file named /tmp/ufm_slurm.log logs all of the actions and errors that occurred during the execution.

This log file can be changed by modifying the log_file_name parameter in /etc/slurm /ufm_slurm.conf.

Last updated: August 27, 2026