Deploying BlueField Software Using BFB from BMC

The following table lists an overview of the steps required to install Ubuntu BFB on your DPU:

mst status -v

MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
BlueField2(rev:1)       /dev/mst/mt41686_pciconf0.1   3b:00.1   mlx5_1          net-ens1f1                0

BlueField2(rev:1)       /dev/mst/mt41686_pciconf0     3b:00.0   mlx5_0          net-ens1f0                0

Ensure RShim is Running on BMC

Display the current setting. Run: 

# cat /dev/rshim<N>/misc | grep DEV_NAME
DEV_NAME        usb-1.0

This output indicates that the RShim service is ready to use. If you do not receive this output:

1. Restart RShim service:

   sudo systemctl restart rshim
   

   
   

2. Verify the current setting again. Run: 

   # cat /dev/rshim<N>/misc | grep DEV_NAME
   

   If DEV_NAME does not appear, then proceed to "RShim driver not loading on DPU with integrated BMC".

Changing Default Credentials Using bf.cfg

Ubuntu users are prompted to change the default password (ubuntu) for the default user (ubuntu) upon first login. Logging in will not be possible even if the login prompt appears until all services are up ("DPU is ready" message appears in /dev/rshim0/misc).

Alternatively, Ubuntu users can provide a unique password that will be applied at the end of the BFB installation. This password would need to be defined in a bf.cfg configuration file. To set the password for the ubuntu user:

1. Create password hash. Run:

   # openssl passwd -1
   Password:
   Verifying - Password:
   $1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1
   

   
   

2. Add the password hash in quotes to the bf.cfg file:

   # vim bf.cfg
   ubuntu_PASSWORD='$1$3B0RIrfX$TlHry93NFUJzg3Nya00rE1'
   

   The bf.cfg file is used with the bfb-install script in the steps that follow.

   Password policy:

   • Minimum password length – 8

   • At least one upper-case letter

   • At least one lower-case letter

   • At least one numerical character

   
   

BFB Installation

To update the software on the NVIDIA® BlueField® device, the BlueField must be booted up without mounting the eMMC flash device. This requires an external boot flow where a BFB (which includes ATF, UEFI, Arm OS, NIC firmware, and initramfs) is pushed from an external host via USB or PCIe. On BlueField devices with an integrated BMC, the USB interface is internally connected to the BMC and is enabled by default. Therefore, you must verify that the RShim driver is running on the BMC. This provides the ability to push a bootstream over the USB interface to perform an external boot.

1. Disable RShim on the host. Run on the host:

   systemctl stop rshim
   systemctl disable rshim
   

2. Enable RShim on the BMC. Run on the BMC shell:

   ipmitool raw 0x32 0x6a 1
   

3. Verify that the RShim service is running and that the /dev/rshim0 device is present. 

   ipmitool raw 0x32 0x69
   

   The expected output for this command is 0x01.

4. Normally, the BFB is first copied to the USB host and then written to the RShim device. However, since the BFB is too big to be stored on the BMC flash or tmpfs, these two steps can be combined, and the image can be written to the RShim device from the remote server directly.

   scp <path_to_bfb> root@<bmc_ip>:/dev/rshim0/boot
   

   This initiates a soft reset on the BlueField-2 and then pushes the bootstream. For Ubuntu BFBs, the eMMC is flashed automatically once the bootstream is pushed. 

   If bf.cfg is needed as part of the boot process, run: 

   cat <path_to_bfb> bf.cfg > new.bfb
   scp <path to new.bfb> root@<bmc_ip>:/dev/rshim0/boot
   

Verify BFB is Installed

After installation of the Ubuntu OS is complete, the following note appears in /dev/rshim0/misc on first boot:

...
INFO[MISC]: Linux up
INFO[MISC]: DPU is ready

"DPU is ready" indicates that all the relevant services are up and users can login the system.

After the installation of the Ubuntu 20.04 BFB, the configuration detailed in the following sections is generated.

BlueField OS image version is stored under /etc/mlnx-release in the DPU.

# cat /etc/mlnx-release
DOCA_2.6.0_BSP_4.6.0_Ubuntu_22.04-<version>

Firmware Upgrade

To upgrade firmware:

1. Set a temporary static IP on the host. Run: 

   sudo ip addr add 192.168.100.1/24 dev tmfifo_net0
   

   
   

2. SSH to your DPU via 192.168.100.2 (preconfigured). The default credentials for Ubuntu are as follows.

   Username	Password
   ubuntu	Set during installation

   For example:

   ssh ubuntu@192.168.100.2
   Password: <unique-password>
   

   
   

3. Upgrade the firmware on the DPU. Run:

   sudo /opt/mellanox/mlnx-fw-updater/mlnx_fw_updater.pl --force-fw-update
   

   Example output: 

   Device #1:
   ----------
   
     Device Type:      BlueField-2
     [...]
     Versions:         Current        Available
        FW             <Old_FW>       <New_FW>
   

   
   

   Important! To apply NVConfig changes, stop here and follow the steps in section "Updating NVConfig Params".

   
   

4. Perform a graceful shutdown and power cycle the host for the changes to take effect.

Updating NVConfig Params

1. Reset the nvconfig params to their default values: 

   # sudo mlxconfig -d /dev/mst/<device-id> -y reset
    
   Reset configuration for device /dev/mst/<device-name>? (y/n) [n] : y
   Applying... Done!
   -I- Please reboot machine to load new configurations.
   

   
   

   To learn the device ID of the DPUs on your setup, run:

   mst start
   mst status -v
   

   Example output:

   MST modules:
   ------------
       MST PCI module is not loaded
       MST PCI configuration module loaded
   PCI devices:
   ------------
   DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA
   BlueField2(rev:1)       /dev/mst/mt41686_pciconf0.1   3b:00.1   mlx5_1          net-ens1f1                0
    
   BlueField2(rev:1)       /dev/mst/mt41686_pciconf0     3b:00.0   mlx5_0          net-ens1f0                0
   
   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0.1   e2:00.1   mlx5_1          net-ens7f1np1             4
   
   BlueField3(rev:1)       /dev/mst/mt41692_pciconf0     e2:00.0   mlx5_0          net-ens7f0np0             4
   

   The device IDs for the BlueField-2 and BlueField-3 DPUs in this example are /dev/mst/mt41686_pciconf0 and /dev/mst/mt41692_pciconf0 respectively.

   
   

2. (Optional) Enable NVMe emulation. Run: 

   sudo mlxconfig -d <device-id> s NVME_EMULATION_ENABLE=1
   

   
   

3. Skip this step if your BlueField DPU is Ethernet only. Please refer to section "Supported Platforms and Interoperability" under the Release Notes to learn your DPU type.

   If you have a VPI DPU, the default link type of the ports will be configured to IB. If you want to change the link type to Ethernet, please run the following configuration:

   sudo mlxconfig -d <device-id> s LINK_TYPE_P1=2 LINK_TYPE_P2=2
   

   
   

4. Perform a graceful shutdown and power cycle the host for the mlxconfig settings to take effect.

Last updated: January 02, 2024