SoC Management Interface (AKA RShim)

The SoC management interface, formerly known as RShim, allows an external agent such as the host CPU or BMC to operate the NVIDIA® BlueField® networking platform (DPU or SuperNIC) and monitor its operational state. This interface allows provisioning of BlueField, resetting Arm cores, and obtaining logs.

Installation and Upgrade

Please refer to section Updating Repo Package on Host Side.

Configuration File

The configuration file for the SoC management interface is located at /etc/rshim.conf and includes the parameters listed in the table below.

Users may control which RShim index maps to which device by following this procedure:

# Uncomment the 'rshim<N>' line to configure the mapping.
#
# device-name pci-device
rshim0      pcie-0000:21:00.2
rshim1      pcie-0000:81:00.2

#
# Ignored devices.
# Uncomment the 'none' line to configure the ignored devices.
#
#none         usb-1-1.4
#none         pcie-lf-0000:84:00.0

systemctl restart rshim

Host-side Interface Configuration

BlueField registers on the host OS a "DMA controller" for BlueField management over PCIe. This can be verified by running the following: 

#  lspci -d 15b3: | grep 'SoC Management Interface'
27:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01)

A special SoC management driver must be installed and run on the host OS to expose the various BlueField management interfaces to the OS. Currently, this driver is named RShim and is automatically installed as part of the DOCA installation. Refer to section "Install RShim on Host" for information on how to obtain and install the host-side SoC management interface driver.

When the SoC management interface driver runs properly on the host side, a sysfs device, /dev/rshim0/*, and a virtual Ethernet interface, tmfifo_net0, become available. The following is an example for querying the status of the SoC management interface driver on the host side:

# systemctl status rshim 
● rshim.service - rshim driver for BlueField SoC
     Loaded: loaded (/lib/systemd/system/rshim.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-05-31 14:57:07 IDT; 1 day 1h ago
       Docs: man:rshim(8)
    Process: 90322 ExecStart=/usr/sbin/rshim $OPTIONS (code=exited, status=0/SUCCESS)
   Main PID: 90323 (rshim)
      Tasks: 11 (limit: 76853)
     Memory: 3.3M
     CGroup: /system.slice/rshim.service
             └─90323 /usr/sbin/rshim
May 31 14:57:07 …  systemd[1]: Starting rshim driver for BlueField SoC...
May 31 14:57:07  … systemd[1]: Started rshim driver for BlueField SoC.
May 31 14:57:07  … rshim[90323]: Probing pcie-0000:a3:00.2(vfio)
May 31 14:57:07  … rshim[90323]: Create rshim pcie-0000:a3:00.2
May 31 14:57:07  … rshim[90323]: rshim pcie-0000:a3:00.2 enable
May 31 14:57:08  … rshim[90323]: rshim0 attached

Virtual Ethernet Interface

On the host, the SoC management interface driver exposes a virtual Ethernet device called tmfifo_net0. This virtual Ethernet can be thought of as a peer-to-peer tunnel connection between the host and the BlueField OS. The BlueField OS also configures a similar device. The BlueField OS's BFB images are customized to configure the BlueField side of this connection with a preset IP of 192.168.100.2/30. It is up to the user to configure the host side of this connection. Configuration procedures vary for different OSs.

The following example configures the host side of tmfifo_net0 with a static IP and enables IPv4-based communication to the BlueField OS:

#  ip addr add dev tmfifo_net0 192.168.100.1/30

Logging in from the host to the BlueField OS is now possible over the virtual Ethernet. For example:

ssh ubuntu@192.168.100.2

SoC Management Interface Driver Support for Multiple BlueFields

Multiple BlueField devices may connect to the same host machine. When the SoC management interface driver is loaded and operating correctly, each BlueField device is expected to have its own device directory on sysfs, /dev/rshim<N>, and a virtual Ethernet device, tmfifo_net<N>. 

The following are some guidelines on how to set up the SoC management virtual Ethernet interfaces properly if multiple BlueField devices are installed in the host system.

There are two methods to manage multiple tmfifo_net interfaces on a Linux platform:

• Using a bridge, with all tmfifo_net<N> interfaces on the bridge – the bridge device bares a single IP address on the host while each BlueField has unique IP in the same subnet as the bridge

• Directly over the individual tmfifo_net<N> – each interface has a unique subnet IP and each BlueField has a corresponding IP per subnet

Whichever method is selected, the host-side tmfifo_net interfaces should have different MAC addresses, which can be:

• Configured using ifconfig. For example:

  $ ifconfig tmfifo_net0 192.168.100.1/24 hw ether 02:02:02:02:02:02
  

• Or saved in configuration via the /udev/rules as can be seen later in this section.

In addition, each Arm-side tmfifo_net interface must have a unique MAC and IP address configuration, as BlueField OS comes uniformly pre-configured with a generic MAC, and 192.168.100.2. The latter must be configured in each BlueField manually or by BlueField customization scripts during BlueField OS installation.

Multi-board Management Example

This example deals with two BlueField devices installed on the same server (the process is similar for more devices). The example assumes that the RShim package has been installed on the host server.

Configuring Management Interface on Host

1. Create a bf_tmfifo interface under /etc/sysconfig/network-scripts. Run: 

   vim /etc/sysconfig/network-scripts/ifcfg-br_tmfifo
   

2. Inside ifcfg-br_tmfifo, insert the following content: 

   DEVICE="br_tmfifo"
   BOOTPROTO="static"
   IPADDR="192.168.100.1"
   NETMASK="255.255.255.0"
   ONBOOT="yes"
   TYPE="Bridge"
   

3. Create a configuration file for the first BlueField, tmfifo_net0. Run: 

   vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0
   

4. Inside ifcfg-tmfifo_net0, insert the following content: 

   DEVICE=tmfifo_net0
   BOOTPROTO=none
   ONBOOT=yes
   NM_CONTROLLED=no
   BRIDGE=br_tmfifo
   

5. Create a configuration file for the second BlueField, tmfifo_net1. Run: 

   DEVICE=tmfifo_net1
   BOOTPROTO=none
   ONBOOT=yes
   NM_CONTROLLED=no
   BRIDGE=br_tmfifo
   

6. Create the rules for the tmfifo_net interfaces. Run: 

   vim /etc/udev/rules.d/91-tmfifo_net.rules
   
   

7. Restart the network for the changes to take effect. Run:

   # /etc/init.d/network restart
   Restarting network (via systemctl):            [  OK  ]
   

Configuring BlueField Side

BlueField devices arrive with the following factory default configurations for tmfifo_net0.

Therefore, if you are working with more than one BlueField, you must change the default MAC and IP addresses.

Updating RShim Network MAC Address

1. Use a Linux console application (e.g. screen or minicom) to log into each BlueField. For example:

   # sudo screen /dev/rshim<0|1>/console 115200
   

2. Create a configuration file for tmfifo_net0 MAC address. Run: 

   # sudo vi /etc/bf.cfg
   

3. Inside bf.cfg, insert the new MAC: 

   NET_RSHIM_MAC=00:1a:ca:ff:ff:03
   

4. Apply the new MAC address. Run: 

   sudo bfcfg
   

5. Repeat this procedure for the second BlueField (using a different MAC address). 

   Arm must be rebooted for this configuration to take effect. It is recommended to update the IP address before you do that to avoid unnecessary reboots.

Updating IP Address

For Ubuntu:

1. Access the file 50-cloud-init.yaml and modify the tmfifo_net0 IP address: 

   sudo vim /etc/netplan/50-cloud-init.yaml
   
                   tmfifo_net0:
                        addresses:
                        - 192.168.100.2/30    ===>>>    192.168.100.3/30
   

2. Reboot the Arm. Run: 

   sudo reboot
   

3. Repeat this procedure for the second BlueField (using a different IP address).

   Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

For CentOS:

1. Access the file ifcfg-tmfifo_net0. Run: 

   # vim /etc/sysconfig/network-scripts/ifcfg-tmfifo_net0
   

2. Modify the value for IPADDR:

   IPADDR=192.168.100.3
   

3. Reboot the Arm. Run:

   reboot
   

   Or perform netplan apply.

4. Repeat this procedure for the second BlueField (using a different IP address).

   Arm must be rebooted for this configuration to take effect. It is recommended to update the MAC address before you do that to avoid unnecessary reboots.

Permanently Changing Arm-side MAC Address

The default MAC address is 00:1a:ca:ff:ff:01. It can be changed using ifconfig or by updating the UEFI variable as follows:

1. Log into Linux from the Arm console.

2. Run:

   $ "ls /sys/firmware/efi/efivars".
   

3. If not mounted, run:

   $ mount -t efivarfs none /sys/firmware/efi/efivars
   $ chattr -i /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
   $ printf "\x07\x00\x00\x00\x00\x1a\xca\xff\xff\x03" > \
     /sys/firmware/efi/efivars/RshimMacAddr-8be4df61-93ca-11d2-aa0d-00e098032b8c
   

The printf command sets the MAC address to 00:1a:ca:ff:ff:03 (the last six bytes of the printf value). Either reboot the device or reload the tmfifo driver for the change to take effect. 

The MAC address can also be updated from the server host side while the Arm-side Linux is running:

1. Enable the configuration. Run: 

   # echo "DISPLAY_LEVEL 1" > /dev/rshim0/misc
   

2. Display the current setting. Run:

   # cat /dev/rshim0/misc
   DISPLAY_LEVEL   1 (0:basic, 1:advanced, 2:log)
   BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
   BOOT_TIMEOUT    300 (seconds)
   DROP_MODE       0 (0:normal, 1:drop)
   SW_RESET        0 (1: reset)
   DEV_NAME        pcie-0000:04:00.2
   DEV_INFO        BlueField-2(Rev 1)
   PEER_MAC        00:1a:ca:ff:ff:01 (rw)
   PXE_ID          0x00000000 (rw)
   VLAN_ID         0 0 (rw)
   

3. Modify the MAC address. Run:

   $ echo "PEER_MAC  xx:xx:xx:xx:xx:xx" > /dev/rshim0/misc
   

SoC Management Interface Features and Functionality

bfb-install –r rshim<N> -b <bfb> [-c bf.cfg]

screen /dev/rshim<N>/console 115200 
minicom -D /dev/rshim<N>/console 

ip addr add dev tmfifo_net<N> 192.168.100.1/30

ssh -6  user@fe80::21a:caff:feff:ff01%tmfifo_net<N>

$ sudo cat /dev/rshim1/misc
DISPLAY_LEVEL   2 (0:basic, 1:advanced, 2:log)
BF_MODE         NIC mode
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    300 (seconds)
USB_TIMEOUT     40 (seconds)
DROP_MODE       0 (0:normal, 1:drop)
SW_RESET        0 (1: reset)
DEV_NAME        pcie-0000:b1:00.2
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         9009D3B600CVAA
UP_TIME         595371(s)
SECURE_NIC_MODE 0 (0:no, 1:yes)
FORCE_CMD       0 (1: send Force command)
---------------------------------------
             Log Messages
---------------------------------------
 INFO[MISC]: Linux up
 INFO[MISC]: DPU is ready

echo "SW_RESERT 1" > /dev/rshim<N>/misc

echo DEV_NAME > /dev/rshim0/misc

Rshim Misc Interface

echo "SW_RESET 1" > /dev/rshim0/misc

BlueField Configuration File

The bf.cfg file contains configuration that can be pushed to customize the installation of the BFB.

See "Customizing BlueField Software Deployment" for more information.

RShim Ownership

The RShim interface may be owned by the BlueField BMC or the host (Windows or Linux). In situations where users do not have access to the host, they would want to transfer RShim ownership to the BMC.

Assuming that /dev/rshim0 is the BlueField requesting ownership over the RShim interface, ownership may be transferred to the BlueField BMC by running the following command from the BMC console:

1. Confirm RShim is not attached to BMC:

   root@dpu-bmc:~# systemctl status rshim
   

   This show messages like another backend already attached or rshim0 entering drop mode. 

   If RShim is already attached or RShim is not in drop mode, the ownership transfer command (-F) will fail.

2. Create a directory for the RShim systemd override file if it does not already exist:

   root@dpu-bmc:~# mkdir -p /etc/systemd/system/rshim.service.d
   

3. Create the override file: 

   root@dpu-bmc:~# cat > /etc/systemd/system/rshim.service.d/override.conf
   [Service]
   Environment="OPTIONS=-F"
   

4. Press Ctrl-D to save the file.

5. Reload the systemd manager configuration and restart RShim service.

   root@dpu-bmc:~# systemctl daemon-reload
   root@dpu-bmc:~# systemctl restart rshim
   root@dpu-bmc:~# systemctl status rshim
   … …
   Dec 10 20:58:57 dpu-bmc rshim[20561]: rshim0 received ownership transfer ack
   Dec 10 20:59:00 dpu-bmc rshim[20561]: rshim0 regained ownership successfully
   

   RShim ownership is transferred to BMC.

6. Clean up the systemd override file:

   root@dpu-bmc:~# rm /etc/systemd/system/rshim.service.d/override.conf
   root@dpu-bmc:~# systemctl daemon-reload
   

Controlling SoC management interface could be done also from platform bmc using NC-SI OEM command over I2C.

rshim interface could be enabled, disabled or locked:

• Enabled - rshim over PCIE is supported and ready to be used

• Locked - rshim over PCIE is supported, but currently cannot be used. It could be re-activated using enable command, without a need to perform any reset

• Disabled - rshim over PCIE is not supported, needed to re-enable it and power cycle the host x86 

Set Host PCIe rshim access to BlueField DPU CPU (Command=0x12, Parameter=0x19) 

The Set Host PCIe rshim access to BlueField DPU CPU command allows a trusted platform bmc to configure the BlueField DPU enablement state of the external Host PCIe rshim access to the embedded CPU.

The Set Host PCIe rshim access to BlueField DPU CPU command is a package command.

The action requires to reboot BlueField DPU to take effect.

Set Host PCIe rshim access to BlueField DPU CPU Format

Set Host PCIe rshim access to BlueField DPU CPU Command Parameters

Set Host PCIe rshim access to BlueField DPU CPU Response

BlueField DPU always receive and respond to Set Host PCIe rshim access to BlueField DPU CPU command when the package ID matches.

If the command is received from an untrusted platform bmc, the command will fail with reason code 0x7FFF (Unsupported command).

Set Host rshim Access to BlueField DPU CPU Response Format

Set Dynamic Host PCIe rshim access to BlueField DPU CPU (Command=0x12, Parameter=0x1B) 

The Set Dynamic Host PCIe rshim access to BlueField DPU CPU command allows a trusted platform bmc to configure the BlueField DPU runtime enablement state of the external Host PCIe rshim access to the embedded CPU.

This command is only available when the non-volatile setting of the Host PCIe rshim access is set to either Enabled or Locked states (see definition in section "Set Host PCIe rshim access to BlueField DPU CPU").

The setting in this command is volatile. Upon device reset, the value of Host_RT_Access_State defaults to the non-volatile configuration (as defined in section "Set Host PCIe rshim access to BlueField DPU CPU").

Set Dynamic Host PCIe rshim access Access to BlueField DPU CPU Format

Set Dynamic Host PCIe rshim access Access to BlueField DPU CPU Command Parameters

Set Dynamic Host rshim Access to BlueField DPU CPU Response

BlueField DPU only receive and respond to Set Dynamic Host PCIe rshim access to BlueField DPU CPU Command when the package ID matches and the non-volatile setting of the Host PCIe rshim access is one of Enabled or Locked states.

If this command is received when the non-volatile setting is set to Disabled, this command shall fail with response code 0x0002 (command unavailable).

If the command is received from an untrusted platform bmc, the command will fail with reason code 0x7FFF (unsupported command).

The command enables toggling between enabled to locked without BlueField DPU reset

Set Dynamic Host rshim Access to BlueField DPU CPU Response Format

Get Host PCIe rshim access to BlueField DPU CPU (Command=0x13, Parameter=0x19) 

The Get Host PCIe rshim access to BlueField DPU CPU command allows the platform bmc (platform bmc) to query the BlueField DPU about the enablement state of the external Host PCIe rshim access to the embedded CPU.

The Get Host PCIe rshim access to BlueField DPU CPU command is a package command.

Get Host PCIe rshim access to BlueField DPU CPU Format

Get Host PCIe rshim access to BlueField DPU CPU Response

BlueField DPU always receive and respond to Get Host PCIe rshim access to BlueField DPU CPU Command when the package ID matches.

Get Host PCIe rshim access to BlueField DPU CPU Response

Get Host PCIe rshim access to BlueField DPU CPU Response Format

Last updated: April 21, 2025