NVIDIA SM

NVIDIA SM is an InfiniBand-compliant Subnet Manager (SM) and Subnet Administrator provided as the opensm executable. It runs on top of the NVIDIA OFED stack and initializes, configures, and monitors an InfiniBand subnet.

One SM must be running for each InfiniBand subnet. OpenSM attaches to one local InfiniBand port and manages only the fabric connected to that port.

ConnectX-8: Before running OpenSM or InfiniBand management tools on a ConnectX-8 host, create an SMI interface as described in Creating an SMI Interface on ConnectX-8.

Quick Start

Before You Begin

Before starting OpenSM:

• Ensure that the NVIDIA OFED stack and OpenSM are installed on the host.

• Run the commands with sufficient privileges.

• Confirm that another active SM is not already managing the same subnet unless you are intentionally configuring an additional SM for failover.

• Identify the InfiniBand device or port on which OpenSM will run.

• On ConnectX-8 hosts, create an SMI interface before starting OpenSM.

By default:

• OpenSM writes its log to /var/log/opensm.log.

• OpenSM stores persistent data in /var/cache/opensm.

• OpenSM creates temporary files such as opensm-subnet.lst, opensm.fdbs, and opensm.mcfdbs under /var/log, unless OSM_TMP_DIR is set.

Identify the InfiniBand Device

Run:

ibstat

Select the InfiniBand device or port that is connected to the subnet that OpenSM must manage.

ConnectX-8 only: Continue with the next section to create an SMI interface. For other adapters, continue with Start OpenSM.

Creating an SMI Interface on ConnectX-8

To run OpenSM or InfiniBand management tools such as ibnetdiscover on a host equipped with a ConnectX-8 adapter, first create an SMI interface for the selected InfiniBand device. This requirement applies to standard InfiniBand fabrics and is not limited to multiplane-subnet deployments.

1. Identify the available InfiniBand devices:

   Bash

   ibstat
   

2. Select the device on which OpenSM will run. For example, to create an SMI interface named smi1 for mlx5_0, run:

   Bash

   sudo /opt/mellanox/iproute2/sbin/rdma dev add smi1 type SMI parent mlx5_0
   

   Replace mlx5_0 with the name of the selected device.

3. Verify that the SMI interface was created:

   Bash

   ibstat
   

   The new smi1 interface should appear with its associated ports. Example:

   CA 'smi1'
           CA type: MT4133
           Number of ports: 4
           Firmware version: 82.48.1000
           Hardware version: 0
           Node GUID: 0x70792d0300942df0
           System image GUID: 0x70792d0300942df0
           Port 1:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 51
                   LMC: 0
                   SM lid: 51
                   Capability mask: 0xa750e848
                   Port GUID: 0x70792d0300942df0
                   Link layer: InfiniBand
           Port 2:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 51
                   LMC: 0
                   SM lid: 51
                   Capability mask: 0xa750e848
                   Port GUID: 0x70792d0300942df0
                   Link layer: InfiniBand
           Port 3:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 51
                   LMC: 0
                   SM lid: 51
                   Capability mask: 0xa750e848
                   Port GUID: 0x70792d0300942df0
                   Link layer: InfiniBand
           Port 4:
                   State: Active
                   Physical state: LinkUp
                   Rate: 200
                   Base lid: 51
                   LMC: 0
                   SM lid: 51
                   Capability mask: 0xa750e848
                   Port GUID: 0x70792d0300942df0
                   Link layer: InfiniBand
   

Start OpenSM

To start OpenSM as a service, run:

service opensmd start

OpenSM can also be run directly with command-line options:

opensm [OPTIONS]

To display the supported options, run:

opensm --help

If no port is specified, OpenSM selects the first suitable available port. On hosts with multiple InfiniBand ports or fabrics, explicitly select the intended port to avoid managing the wrong subnet.

Verify Operation

After starting OpenSM:

1. Open /var/log/opensm.log.

2. Confirm that the log contains the following message:

   SUBNET UP
   

3. Run ibstat and confirm that the managed port is active and reports the expected SM LID.

Treat errors in /var/log/opensm.log as indicators of InfiniBand fabric health or configuration problems. OpenSM exits when it encounters a fatal, nonrecoverable error.

Operating OpenSM

Reload Configuration

Send OpenSM a SIGHUP signal by using the HUP signal name to trigger a new heavy sweep and reload supported configuration changes:

pkill -HUP opensm

The following table summarizes common change behavior:

OpenSM also supports SIGUSR1 to reopen /var/log/opensm.log, for example during log rotation.

Environment Variables

Configuring Partitions

OpenSM uses partitions, identified by PKey values, to divide an InfiniBand fabric into logical groups. By default, OpenSM reads the partition configuration from /etc/opensm/partitions.conf. Use --Pconfig or -P to select a different file.

The default partition:

• Is created unconditionally, even if no partition configuration file exists.

• Uses the PKey value 0x7fff.

• Grants full membership to the port on which OpenSM runs.

• Grants limited membership to other end ports unless configured otherwise.

• Cannot be removed.

Configure a Partition

1. Edit /etc/opensm/partitions.conf.

2. Define the partition name, PKey, optional properties, and members.

3. Save the file.

4. Apply membership-only changes with:

   Bash

   pkill -HUP opensm
   

5. Restart OpenSM if you changed existing partition properties such as ipoib, sl, scope, rate, mtu, or multicast-group settings.

Minimal Examples

Default=0x7fff : ALL, SELF=full ;
Default=0x7fff : ALL, ALL_SWITCHES=full, SELF=full ;
NewPartition=0x300 : SELF=full, ALL=limited ;

The following definition is equivalent to the behavior of OpenSM before the partition manager was introduced:

Default=0x7fff,ipoib:ALL=full;

Partition File Syntax

Comments begin with # and continue to the end of the line. Whitespace is allowed around delimiters.

General form:

<PartitionDefinition> : <PartitionProperties> ;

Partition definition:

[PartitionName][=PKey][,indx0][,ipoib_bc_flags][,defmember=full|limited]

Port member:

<PortGUID>[=full|limited|both]

Multicast group:

mgid=<gid>[,<mgroup_flag>]*

Useful member keywords:

Notes:

• Port GUIDs can be hexadecimal or decimal. Hexadecimal values should begin with 0x.

• If no membership type is specified, limited membership is assumed.

• A partition name does not have to be unique, but its PKey must be unique.

• Definitions that repeat the same explicit PKey are merged; the first partition name is used.

• Values for rate, mtu, and scope follow the InfiniBand specification. For example, mtu=5 represents a 4 KB MTU.

Extended Partition Example

ShareIO=0x80,defmember=full : 0x123451,0x123452;
ShareIO=0x80 : 0x123453,0x123454,0x123455=full;
ShareIO=0x80,defmember=limited : 0x123456,0x123457,0x123458=full;

Default=0x7fff,ipoib:
mgid=ff12:401b::0707,sl=1
mgid=ff12:601b::16
mgid=ff12:401b::16
mgid=ff12:601b::2
mgid=ff12::1,sl=1,Q_Key=0xDEADBEEF,rate=3,mtu=2
ALL=full;

Selecting a Routing Engine

OpenSM supports several routing engines. Choose the engine that matches the fabric topology and operational requirements.

LASH routing is not supported.

Min Hop

Min Hop is used by default when no routing engine is specified. It calculates minimum-hop paths and balances LIDs across links with equal path length.

To exclude specific ports from link-load equalization, use:

-i <equalize-ignore-guids-file>
-ignore-guids <equalize-ignore-guids-file>

UPDN

UPDN prevents credit-loop deadlocks in fabrics whose topology contains loops. It ranks switches and restricts routing so that paths do not transition upward after moving downward.

Enable UPDN with:

-R updn

To supply root switches explicitly, use:

-a <root_guid_file>

The <root_guid_file> must contain one root-switch GUID per line. Lines with invalid values are ignored. If <root_guid_file> is not provided, OpenSM attempts to detect root switches automatically.

Fat-tree

The fat-tree engine is intended for symmetrical or nearly symmetrical fat-tree fabrics. It optimizes routing for shift-style communication patterns while avoiding credit loops.

Enable it with:

-R ftree

Without <root_guid_file>, the topology must be a pure fat tree. In general:

• The tree rank must be between 2 and 8.

• Switches at the same rank must have consistent upward and downward port-group structure.

• Compute nodes must be located at the same tree rank.

When <root_guid_file> is supplied, non-compute nodes can exist at other ranks, but compute nodes must remain at the same rank.

A topology that does not meet the requirements causes OpenSM to fall back to Min Hop. A link failure can also trigger fallback if the resulting topology is no longer a valid fat tree.

Routing Between Non-Compute Nodes

Use -G or --io_guid_file to identify non-compute nodes that require connectivity across tree levels. Use -H or --max_reverse_hops to control how many counter-stream hops are allowed.

Use reverse hops only for low-bandwidth connectivity such as high-availability traffic. Counter-stream routes can introduce credit loops and are unsuitable for high-bandwidth traffic.

Fat-tree routing does not support LMC > 0; OpenSM uses the default routing engine if this expression is configured.

DOR

Dimension Order Routing uses shortest paths while selecting links according to a consistent dimension ordering. Use it only when the fabric is cabled as a mesh or hypercube and the same ports represent the same dimensions throughout the fabric.

Enable DOR with:

-R dor

Torus-2QoS

Advanced: Torus-2QoS is intended for large 2D or 3D torus fabrics.

The torus-2QoS routing engine provides:

• Credit-loop-free routing.

• Two QoS levels when the switches support eight data VLs.

• Routing around a single failed switch or multiple failed links, subject to topology constraints.

• Stable path SL values for supported failure scenarios.

Always enable QoS when using torus-2QoS:

-Q

The torus-2QoS engine controls SL-to-VL mappings. Parameters such as qos_sl2vl and qos_swe_sl2vl are ignored when torus-2QoS is active, and OpenSM logs a warning.

Torus Topology Configuration

The torus-2QoS.conf file defines the topology and seed links. Blank lines and lines beginning with # are ignored.

Topology declaration:

[torus|mesh] x_radix[m|M|t|T] y_radix[m|M|t|T] z_radix[m|M|t|T]

Seed-link keywords:

xp_link <sw0_GUID> <sw1_GUID>
yp_link <sw0_GUID> <sw1_GUID>
zp_link <sw0_GUID> <sw1_GUID>
xm_link <sw0_GUID> <sw1_GUID>
ym_link <sw0_GUID> <sw1_GUID>
zm_link <sw0_GUID> <sw1_GUID>

Dateline parameters:

x_dateline <position>
y_dateline <position>
z_dateline <position>

To begin an additional seed definition, use:

next_seed

Other useful parameters:

• portgroup_max_ports <max_ports> sets the maximum number of parallel inter-switch links and host ports per switch. The default is 16.

• port_order <p1> <p2> ... controls the order in which destination CA ports are visited when routes are assigned.

Example:

torus 1 4 5
yp_link 0x200000 0x200005
ym_link 0x200000 0x20000f
zp_link 0x200000 0x200001
next_seed
yp_link 0x20000b 0x200010
ym_link 0x20000b 0x200006
zp_link 0x20000b 0x20000c
y_dateline -2
z_dateline -1
port_order 7 10 8 11 9 12 25 28 26 29 27 30

Operational Considerations

Applications running on a torus fabric must obtain the correct path SL. Applications that use rdma_cm do this automatically.

The torus-2QoS engine can preserve path SL values across supported link or switch failures and SM failover when all OpenSM instances use the same dateline configuration. If torus-2QoS cannot generate credit-loop-free routes, it logs a warning and refuses to route the fabric.

Use ibdmchk with data collected by ibdiagnet -vlr to validate that the torus fabric is routed without credit loops.

Effect of Topology Changes

When a link is added or removed, OpenSM changes only routes that must be updated. A route must change when its output port is no longer active or no longer represents a minimum-hop path.

The file-based routing engine does not recalculate routes after topology changes. It reloads forwarding tables from the configured file, skips nonexistent switches, and does not revalidate disconnected-node LIDs. Multicast routing is not affected by the file-based unicast engine.

Unicast Routing Cache

The unicast routing cache prevents unnecessary route recalculation during heavy sweeps when no relevant topology change occurred. It can also preserve routing when CAs, routers, or leaf switches go down and return without requiring a full recalculation.

Offsweep Balancing

Offsweep balancing allows Min Hop, DOR, and UPDN to rebalance routes during idle time between sweeps.

offsweep_balancing_enabled TRUE|FALSE
offsweep_balancing_window 180

• offsweep_balancing_enabled is FALSE by default.

• offsweep_balancing_window defines how many seconds OpenSM waits after a sweep before beginning rebalancing. The default is 180 seconds.

Configuring Routing Chains

Advanced: Routing chains allow different routing engines to configure different portions of the same fabric.

A routing chain is processed in sequence. If a node belongs to more than one topology, the last applicable routing step determines its final routes.

Configuration Workflow

1. Define port groups.

2. Define topologies based on those port groups.

3. Create optional configuration files for individual routing engines.

4. Define the routing-chain policy.

Port Group Policy

Set the pgrp_policy_file parameter in the main OpenSM configuration file:

pgrp_policy_file /etc/opensm/conf/port_groups_policy_file

Each group begins with the port-group keyword and ends with the end-port-group keyword:

port-group
name: grp1
port-guid: 0x281, 0x282, 0x283
end-port-group

Common qualifiers:

The port-guid qualifier supports individual ports, lists, and ranges. Examples include 0x283@3, 0x286@1/5/7, and 0x289@2-5/7/9-13/18.

The port-name qualifier selects HCA ports by node description. It requires the hostname field and can include the optional port and hca_idx fields. Use hostname=<name>; port=<number>; hca_idx=<number>. Separate multiple selectors with commas. For example:

port-name: hostname=kuku; port=2; hca_idx=1, hostname=host1; port=3, hostname=host2

When referring to these fields in prose, format them as hostname/port/hca_idx.

The port-regexp qualifier can include an optional physical-port selector after a colon. For example, port-regexp: SW:3 selects physical port 3 on matching nodes.

Predefined groups:

• ALL

• ALL_SWITCHES

• ALL_CAS

• ALL_ROUTERS

These names are reserved and cannot be redefined in the policy file.

Example:

port-group
name: grp1
port-guid: 0x281, 0x282, 0x283
end-port-group

port-group
name: grp2
port-guid-range: 0x282-0x286
port-name: hostname=server1; port=1
end-port-group

port-group
name: grp3
subtract-rule: grp1, grp2
end-port-group

Topology Policy

Set the topo_policy_file parameter:

topo_policy_file /etc/opensm/conf/topo_policy_file.cfg

Each topology requires the id, sw-grp, and hca-grp qualifiers:

topology
id: 1
sw-grp: fabric_switches
hca-grp: fabric_hosts
end-topology

Per-Engine Configuration

A routing step can reference a dedicated OpenSM configuration file by using the config qualifier. Unspecified values are inherited from the main configuration file.

The following parameters take effect only from the main OpenSM configuration file:

• QoS parameters matching qos_*.

• lmc.

• routing_engine.

Routing Chain Policy

Set the rch_policy_file parameter:

rch_policy_file /etc/opensm/conf/chains_policy_file

Each routing step begins with the unicast-step keyword and ends with the end-unicast-step keyword:

unicast-step
id: 1
engine: minhop
topology: 0
path-bit: 0
end-unicast-step

Qualifiers:

The first routing engine must cover the entire fabric by using topology: 0, and its path-bit value must be 0.

Routing Chain Dump Files

When the applicable log_flags value is enabled, each engine writes its own dump files, including:

• opensm-lid-matrix.dump

• opensm-lfts.dump

• opensm.fdbs

• opensm-subnet.lst

The engine ID and name are added to each filename, for example:

opensm-lid-matrix.2.minhop.dump
opensm.fdbs.3.ftree
opensm-subnet.4.updn.lst

The sl2vl and mcfdbs files are written once for the complete fabric rather than once per engine.

Configuring Quality of Service

Enable QoS with -Q or --qos. During initialization and every heavy sweep, OpenSM parses the QoS policy, matches requests to rules, and applies the corresponding QoS level.

Choose the policy type that fits the deployment:

Advanced QoS Policy

An advanced policy can contain:

• port-groups: Reusable groups of source or destination ports.

• qos-setup: Parsed but not supported in NVIDIA OFED; configure SL-to-VL and VL arbitration in the OpenSM options file instead.

• qos-levels: Definitions containing sl and optional mtu-limit, rate-limit, pkey, and packet-life parameters.

• qos-match-rules: Ordered matching rules. The first matching rule wins.

A QoS level named DEFAULT is required.

Minimal policy:

qos-levels
    qos-level
        name: DEFAULT
        sl: 0
    end-qos-level
end-qos-levels

Example:

port-groups
    port-group
        name: Storage
        port-guid: 0x10000000000001, 0x10000000000005-0x1000000000FFFA
    end-port-group
end-port-groups

qos-levels
    qos-level
        name: DEFAULT
        sl: 0
    end-qos-level

    qos-level
        name: StorageLevel
        sl: 1
        mtu-limit: 4
        rate-limit: 5
        pkey: 0x1234
        packet-life: 8
    end-qos-level
end-qos-levels

qos-match-rules
    qos-match-rule
        use: Storage targets
        destination: Storage
        service-id: 0x10000000000001-0x10000000000FFF
        qos-level-name: StorageLevel
    end-qos-match-rule
end-qos-match-rules

Policy syntax rules:

• Leading and trailing whitespace and blank lines are ignored.

• # begins a comment.

• Section-opening keywords require corresponding closing keywords.

• Rules are evaluated in order; the first matching rule wins.

• A DEFAULT QoS level is mandatory.

Simple QoS Policy

Simple QoS rules map ULPs or request attributes directly to an SL.

Minimal policy:

qos-ulps
    default : 0
end-qos-ulps

Example:

qos-ulps
default                              : 0
sdp, port-num 30000                  : 0
sdp, port-num 10000-20000            : 0
sdp                                  : 1
rds                                  : 2
ipoib, pkey 0x0001                   : 0
ipoib                                : 4
any, service-id 0x6234               : 6
any, pkey 0x0ABC                     : 6
srp, target-port-guid 0x1234         : 5
any, target-port-guid 0x0ABC-0xFFFFF : 6
end-qos-ulps

Advanced qos-match-rules take precedence over qos-ulps rules. Within each section, rules are processed in order.

IPoIB

The default IPoIB pkey is 0x7fff. The following rules are equivalent:

ipoib : <SL>
ipoib, pkey 0x7fff : <SL>
any, pkey 0x7fff : <SL>

SRP

SRP is commonly matched by the target-port-guid field:

srp, target-port-guid 0x1234 : <SL>
any, target-port-guid 0x1234 : <SL>

Place broad target-port-guid rules near the end of the rule list so they do not unintentionally match other ULPs.

MPI

OpenSM does not force an SL for MPI traffic. Configure the SL through the MPI environment or launch command.

SL-to-VL Mapping and VL Arbitration

Configure SL-to-VL mappings and VL arbitration in the OpenSM options file. Separate parameter sets exist for:

• Channel Adapters: qos_ca_*

• Routers: qos_rtr_*

• Switch port 0: qos_sw0_*

• Switch external ports: qos_swe_*

Example defaults for CAs and switch external ports:

qos_ca_max_vls 15
qos_ca_high_limit 0
qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_ca_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
qos_swe_max_vls 15
qos_swe_high_limit 0
qos_swe_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0
qos_swe_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4
qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

VL arbitration entries use VL:weight pairs. The weight value is measured in 64-byte credits. A weight value of 0 skips the entry.

The qos_<type>_high_limit parameter controls how much high-priority traffic can be sent before low-priority traffic receives an opportunity. The byte limit is high_limit × 4 KB.

• 255 means no byte limit and can starve low-priority VLs.

• 0 allows one high-priority packet before low-priority traffic receives an opportunity.

For a 4 KB MTU, one packet consumes 64 credits. Use weight values that are multiples of 64 for effective arbitration of 4 KB packets.

Example:

qos_ca_max_vls 15
qos_ca_high_limit 6
qos_ca_vlarb_high 0:4
qos_ca_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64
qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7
qos_swe_max_vls 15
qos_swe_high_limit 6
qos_swe_vlarb_high 0:4
qos_swe_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64
qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7

In this example, eight VLs—VL0 through VL7—are configured in the subnet. VL0 is a high-priority VL limited to 6 × 4 KB, or 24 KB, per transmission burst. This configuration is suitable for traffic that requires low latency and uses a small MTU. The remaining VLs are low-priority VLs with different weights, while VL4 is effectively disabled.

QoS Deployment Examples

HPC: MPI and Lustre

Goals:

• Separate MPI from I/O traffic.

• Allocate at least 70% bandwidth to MPI.

• Give Lustre metadata traffic low latency.

• Allocate at least 30% bandwidth to Lustre storage data.

Assign the MPI SL through the launch command:

mpirun -sl 0

QoS policy:

qos-ulps
default                                      : 0
any, target-port-guid OST1,OST2,OST3,OST4    : 1
any, target-port-guid MDS1,MDS2              : 2
end-qos-ulps

Replace OST* and MDS* with the actual port GUIDs.

OpenSM options:

qos_max_vls 8
qos_high_limit 0
qos_vlarb_high 2:1
qos_vlarb_low 0:96,1:224
qos_sl2vl 0,1,2,3,4,5,6,7,15,15,15,15,15,15,15,15

Two-Tier EDC: IPoIB and SRP

QoS policy:

qos-ulps
default                                  : 0
ipoib                                    : 1
sdp                                      : 1
srp, target-port-guid SRPT1,SRPT2,SRPT3 : 2
end-qos-ulps

Replace SRPT* with the actual SRP target port GUIDs.

OpenSM options:

qos_max_vls 8
qos_high_limit 0
qos_vlarb_high 1:32,2:32
qos_vlarb_low 0:1
qos_sl2vl 0,1,2,3,4,5,6,7,15,15,15,15,15,15,15,15

Three-Tier EDC: IPoIB, RDS, and SRP

QoS policy:

qos-ulps
default                                  : 0
ipoib, pkey 0x8001                       : 1
ipoib, pkey 0x8002                       : 2
rds                                      : 3
srp, target-port-guid SRPT1,SRPT2,SRPT3 : 4
end-qos-ulps

Replace SRPT* with the actual SRP target port GUIDs.

OpenSM options:

qos_max_vls 8
qos_high_limit 0
qos_vlarb_high 1:32,2:96,3:96,4:96
qos_vlarb_low 0:1
qos_sl2vl 0,1,2,3,4,5,6,7,15,15,15,15,15,15,15,15

Partition configuration:

Default=0x7fff,ipoib : ALL=full;
PartA=0x8001,sl=1,ipoib : ALL=full;

Enhanced QoS

Enhanced QoS applies rate limits per SL to physical ports, virtual ports, and port groups. Set enhanced_qos_policy_file to the full path of the policy. Set it to null to disable Enhanced QoS. Standard QoS must also be enabled.

The policy has three sections:

Example syntax:

BW_NAMES
My_bandwidth = 50

BW_RULES
0x2c90000000025 = 5:My_bandwidth,7:My_bandwidth
Port_grp1 = 3:My_bandwidth,9:My_bandwidth

VPORT_BW_RULES
0x2c90000000026 = 5:My_bandwidth,7:My_bandwidth

Special keywords:

• all applies a bandwidth setting to every SL in a rule.

• default applies to GUIDs that do not have an explicit rule.

• A rate-limit value of 0 means unlimited bandwidth.

The enhanced_qos_vport0_unlimit_default_rl parameter controls how virtual port 0 is treated:

• TRUE—unspecified SLs on explicitly listed virtual port 0 GUIDs are unlimited, and unlisted virtual port 0 GUIDs are unlimited on all SLs.

• FALSE—virtual port 0 follows the same rules as other virtual ports.

Send SIGHUP after changing enhanced_qos_vport0_unlimit_default_rl.

Policy example:

BW_NAMES
bw1 = 50
bw2 = 25
bw3 = 15
bw4 = 10

BW_RULES
default = 1:bw1
0x2c90000000025 = 1:bw2

VPORT_BW_RULES
default = all:bw3
0x2c90000000026 = 3:bw4

Verify /var/log/opensm.log after loading an Enhanced QoS policy. Policy parsing failures can produce undefined behavior.

Adaptive Routing, SHIELD, and XDR Multiplane Fabrics

Adaptive Routing Manager supports Adaptive Routing and Self-Healing Networking (SHIELD).

For configuration guidance, refer to Adaptive Routing, HBF, and SHIELD documentation and the HowTo Configure Adaptive Routing and Self-Healing Networking community article.

XDR Multiplane Restrictions

XDR multiplane only: The following restrictions do not apply to standard ConnectX-8 InfiniBand fabrics.

When configuring routing for multiplane subnets in XDR topologies:

• Only the ar_updn routing mode is supported.

• The ar_ftree routing mode is not supported for XDR topologies.

• ar_tree_asymmetric_flow accepts only 0, 1, or 3.

• The customer must provide the root_guid file manually.

IB Router Support

Configure the following OpenSM parameters to enable IB router support:

Security: DOS MAD Prevention

DOS MAD prevention protects host memory by limiting each agent's receive queue. Incoming MADs above the configured threshold are dropped instead of being queued.

Enable DOS MAD Prevention

1. Edit /etc/modprobe.d/mlnx.conf.

2. Set enable_rx_threshold to 1 by adding:

   ib_umad enable_rx_threshold 1
   

The threshold can be updated from user space through libibumad by using umad_update_threshold():

int umad_update_threshold(int fd, int threshold);

Parameters:

• fd: File descriptor associated with the agent receive queue.

• threshold: New receive threshold.

Monitoring and Troubleshooting

OpenSM Log

The default log file is:

/var/log/opensm.log

A successful initialization includes:

SUBNET UP

Review /var/log/opensm.log for:

• Configuration parsing errors.

• Port-selection failures.

• Routing-engine success or fallback messages.

• QoS-policy errors.

• Fatal errors that cause OpenSM to exit.

osmtest

osmtest validates the InfiniBand Subnet Manager and Subnet Administrator. It can create an inventory of nodes, ports, and path records, or compare the current fabric against a saved inventory.

Supported test flows include:

• Multicast compliance.

• Event forwarding.

• Service-record registration.

• RMPP stress.

• Small SA-query stress.

Refer to the osmtest man page for command syntax and test options.

OpenSM Activity Report

OpenSM can write an activity report containing selected operational events. Configure activity_report_subjects with a comma-separated list of subjects:

Activity reference:

Command and File Reference

Common Commands

Common Files

Related Documentation

• Adaptive Routing, HBF, and SHIELD

• HowTo Configure Adaptive Routing and Self-Healing Networking

• OpenSM and osmtest man pages

• NVIDIA OFED documentation

Last updated: August 12, 2026