InfiniBand (IB) Router provides the ability to send traffic between two or more IB subnets thereby potentially expanding the size of the network to over 40k end-ports, enabling separation, fault resilience and connection to different topologies used by different subnets.
For more information on IB router architecture and functionality, please refer to the community post IB Router Architecture and Functionality.
Configuring IB Router
Prerequisites
-
Platform support: IB Router functionality is supported only on Q3400-RA and Q3401-RD systems.
-
Subnet Manager (SM): Ensure Subnet Managers are not running before configuring the switch as an IB router and assigning interfaces to the IB subnets (SWIDs). Complete all IB router configuration and SWID assignments before starting SMs.
-
Limitations:
IPoIB: When IB router profile is configured, IPoIB interfaces are not supported and will remain down.
FNM: External FNM ports are restricted to the default SWID and cannot be assigned to other subnets. Servers should not be connected to the external FNM ports on IB router switches.
NVOS Switch Configuration
-
Configure the system profile to enable IB routing with multiple SWIDs:
admin@nvos:~$ nv action change system profile ib-routing enabled num-of-swids 4 The operation will reset the system configuration and initiate a reboot. The resulting system profile values will be: ib-routing: enabled num-of-swids: 4 adaptive-routing-groups: 1792 Type [y] to confirm the system profile change and reboot. Type [N] to abort. Do you want to continue? [y/N] y Action executing ... System will be rebooted, configurations and system files might be deleted Action executing ...Profile change triggers a factory reset. All existing configurations will be reset to defaults and the system will reboot.
-
After the reboot, verify the system profile configuration:
admin@nvos:~$ nv show system profile operational ----------------------- ----------- adaptive-routing enabled adaptive-routing-groups 1792 breakout-mode enabled ib-routing enabled num-of-swids 4-
Ensure that
ib-routingis set to enabled and thatnum-of-swidsmatches your configured value. -
After the profile change, all IB interfaces are assigned to the default SWID (
infiniband-default). To confirm the IB interface assignment to theinfiniband-defaultSWID, use thenv show interfacecommand.
-
-
Assign each IB interface to the appropriate IB subnet (SWID) according to the topology port mappings.
Use the following command format:admin@nvos:~$ nv set interface <interface-id> link ib-subnet <swid-name>where
<interface-id>is an interface name or range, and<swid-name>is the target SWID.
Example:admin@nvos:~$ nv set interface sw3-4p1-2 link ib-subnet infiniband-1 admin@nvos:~$ nv config apply -y Warning: IB interface configuration is changed. In order to apply configuration the port will be restarted.Ensure Subnet Managers are not running before assigning interfaces to SWIDs. SWID assignments must be completed prior to SM configuration.
Changing SWID assignments while Subnet Managers are running may cause inconsistent SM state requiring a reboot of the router. -
Verify the SWID (IB subnet) assignment for IB interfaces:
admin@nvos:~$ nv show interface Interface State Speed MTU Type IB Speed IB Subnet Logical State Physical State --------- ----- ----- ----- ----- -------- ------------------ ------------- -------------- eth0 up 1G 1500 eth eth1 up 1G 1500 eth fnm1 up 800G 4096 fnm xdr infiniband-default Active LinkUp sw1p1 up 800G 4096 ib xdr infiniband-default Active LinkUp sw1p2 up 800G 4096 ib xdr infiniband-default Active LinkUp sw2p1 up 800G 4096 ib xdr infiniband-default Active LinkUp sw2p2 up 800G 4096 ib xdr infiniband-default Active LinkUp sw3p1 up 800G 4096 ib xdr infiniband-1 Active LinkUp sw3p2 up 800G 4096 ib xdr infiniband-1 Active LinkUp sw4p1 up 800G 4096 ib xdr infiniband-1 Active LinkUp sw4p2 up 800G 4096 ib xdr infiniband-1 Active LinkUp ...
Verify IB router
After configuring both the IB router and starting Subnet Managers, verify the routing table:
admin@nvos:~$ nv show ib router routing-table
IB Subnet Subnet Prefix Valid
------------------ ----------------------- -------
infiniband-default fe:c0:00:00:00:00:00:01 true
infiniband-1 fe:c0:00:00:00:00:00:02 true
infiniband-2 00:00:00:00:00:00:00:00 false
infiniband-3 00:00:00:00:00:00:00:00 false
To view the router counters:
admin@nvos:~$ nv show ib router counters
Counter infiniband-default infiniband-1 infiniband-2 infiniband-3
---------------------------- ------------------ ------------ ------------ ------------
in-bytes 0 0 0 0
in-pkts 0 0 0 0
in-drops 0 0 0 0
out-bytes 0 0 0 0
out-pkts 0 0 0 0
out-drops 0 0 0 0
out-errors 0 0 0 0
out-unicast-pkts 0 0 0 0
out-multicast-pkts 0 0 0 0
in-unicast-pkts 0 0 0 0
in-multicast-pkts 0 0 0 0
buffer-overrun-errors 0 0 0 0
out-wait 0 0 0 0
carrier-down-count 0 0 0 0
error-recovery 0 0 0 0
port-rcv-switch-relay-errors 0 0 0 0
port-rcv-constraint-errors 0 0 0 0
local-integrity-errors 0 0 0 0
IB Router Commands
Last updated: