IB Router | NVOS IB

InfiniBand (IB) Router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation, fault resilience and connection to different topologies used by different subnets.

For more information on IB router architecture and functionality, please refer to the community post IB Router Architecture and Functionality.

Configuring IB Router

Prerequisites

Platform support: IB Router functionality is supported only on Q3400-RA and Q3401-RD systems.
Subnet Manager (SM): Ensure Subnet Managers are not running before configuring the switch as an IB router and assigning interfaces to the IB subnets (SWIDs). Complete all IB router configuration and SWID assignments before starting SMs.
Limitations:

IPoIB: When IB router profile is configured, IPoIB interfaces are not supported and will remain down.

FNM: External FNM ports are restricted to the default SWID and cannot be assigned to other subnets. Servers should not be connected to the external FNM ports on IB router switches.

NVOS Switch Configuration

Configure the system profile to enable IB routing with multiple SWIDs:

admin@nvos:~$ nv action change system profile ib-routing enabled num-of-swids 2
The operation will reset the system configuration and initiate a reboot.
Type [y] to confirm the system profile change and reboot.
Type [N] to abort.
Do you want to continue? [y/N] y
Action executing ...
System will be rebooted, configurations and system files might be deleted
Action executing ...

Profile change triggers a factory reset. All existing configurations will be reset to defaults and the system will reboot.

After the reboot, verify the system profile configuration:
```
admin@nvos:~$ nv show system profile
                         operational
-----------------------  -----------
adaptive-routing         enabled
adaptive-routing-groups  1792
breakout-mode            enabled
ib-routing               enabled
num-of-swids             2
```
- Ensure that ib-routing is set to enabled and that num-of-swids matches your configured value.
- After the profile change, all IB interfaces are assigned to the default SWID (infiniband-default). To confirm the IB interface assignment to the infiniband-default SWID, use the nv show interface command.
Assign each IB interface to the appropriate IB subnet (SWID) according to the topology port mappings.
Use the following command format:
```
admin@nvos:~$ nv set interface <interface-id> link ib-subnet <swid-name>
```
where <interface-id> is an interface name or range, and <swid-name> is the target SWID.

Required workflow for new SWID mapping to take effect (Beta):
```
admin@nvos:~$ nv set interface <interface-id> link ib-subnet <swid-name>
admin@nvos:~$ nv config apply -y
admin@nvos:~$ nv config save
admin@nvos:~$ nv action reboot system
```
Example:
```
admin@nvos:~$ nv set interface sw3-4p1-2 link ib-subnet infiniband-1
admin@nvos:~$ nv config apply -y
Warning: IB interface configuration is changed. In order to apply configuration the port will be restarted.
admin@nvos:~$ nv config save
admin@nvos:~$ nv action reboot system
Action executing ...
Performing reboot cold...
Action executing ...
Disconnecting from NVOS, system is offline during reboot
```
Known limitation (Beta)

Due to beta-version limitation, SWID changes take effect only after reboot with saved configuration.
Because nv show interface displays operational state only, it may continue to show the previous SWID until the system is rebooted.
After saving the configuration and rebooting, nv show interface will reflect the new SWID.

Ensure Subnet Managers are not running before assigning interfaces to SWIDs. SWID assignments must be completed prior to SM configuration.
Changing SWID assignments while Subnet Managers are running may cause inconsistent SM state requiring a reboot of the router.

Verify the SWID (IB subnet) assignment for IB interfaces:

admin@nvos:~$ nv show interface
Interface  State  Speed  MTU    Type   IB Speed  IB Subnet           Logical State  Physical State
---------  -----  -----  -----  -----  --------  ------------------  -------------  --------------
eth0       up     1G     1500   eth                                                                
eth1       up     1G     1500   eth                                                                
fnm1       up     800G   4096   fnm    xdr       infiniband-default  Active         LinkUp
sw1p1      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
sw1p2      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
sw2p1      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
sw2p2      up     800G   4096   ib     xdr       infiniband-default  Active         LinkUp
sw3p1      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
sw3p2      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
sw4p1      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
sw4p2      up     800G   4096   ib     xdr       infiniband-1        Active         LinkUp
...

Verify Routing Table

After configuring both the IB router and starting Subnet Managers, verify the routing table:

admin@nvos:~$ nv show ib router routing-table
IB Subnet           Subnet Prefix  Valid
------------------  -------------  -----
infiniband-default  1              true
infiniband-1        2              true

A value of true indicates that the routing entry is active and that traffic can be routed to and from this subnet.

IB Router Commands

IB Router Commands

Last updated: April 28, 2026