Networking Solutions
Networking Solutions
Breadcrumbs

HowTo Configure NVIDIA BlueField-3 to NIC Mode on VMware vSphere 8.0

Created on Mar 12, 2024


Introduction

This guide outlines configuring the NVIDIA® BlueField®-3 DPU to NIC Mode on VMware ESXi version 8.0.x.

References

Hardware and Software Requirements

  • A server platform with an adapter card utilizing an NVIDIA BlueField-3 device

  • Administrator privileges are necessary for installation on the target machine

Setup

The setup involves an ESXi 8.0 server with one or more NVIDIA BlueField-3 adapter cards.

The setup includes:

VMware ESXi server, vSphere Cluster and vCenter install and configuration is out of the scope of this post.

Option 1 - Using the BIOS Menu on DELL and Lenovo Servers

On a Dell server:

    1. Enter the server's BIOS menu.

    2. Click Device Settings: dell 750 01.png

    3. Select the desired BlueField device(s). BF3 Dell BF mode 02.png

    4. Verify that the Chip Type is a BlueField device (BlueField-3) and the VMware Distributed Services Engine (DPU) is Disabled.
      Click BlueField Internal Cpu Configuration. dell 750 02.png

    5. Set Internal Cpu Offload Engine setting to Disable (disable is NIC mode). dell 750 03.png

    6. Repeat steps b-d for all necessary BlueField devices in the server.

    7. Save BIOS settings.

    8. Power cycle the server.

















On a Lenovo server:

    1. Enter the server's BIOS menu.

    2. Select the desired BlueField device(s). 00.png 01.png

    3. Verify that the Chip Type is a BlueField device (e.g., BlueField-3) and the VMware Distributed Services Engine (DPU) is Disabled. 02.png

    4. Click BlueField Internal Cpu Configuration. 03.png

    5. Set Internal Cpu Offload Engine setting to Disable (i.e., NIC mode). 04.png

    6. Repeat steps b-d for all necessary BlueField devices in the server.

    7. Save BIOS settings.

    8. Power cycle the server.

Option 2 - Using the ESXi Server Console

  1. Enable SSH from the vSphere Client to the desired ESXi server.

  2. Access the remote ESXi Shell.

  3. Enter Maintenance Mode with ESXCLI.

  4. Install NVIDIA Firmware Tools (MFT).

  5. Locate the physical NVIDIA network device.

    ESXi console

    [root@qa-esxio-lnv-06:~] lspci |grep -i mel

    Example output:

    ESXi console

    0000:38:00.0 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7]
0000:38:00.1 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7]
0000:38:00.2 Ethernet controller: Mellanox Technologies MT42822 Family  [BlueField Auxiliary Comm Channel]
0000:38:00.3 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface
0000:a8:00.0 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7]
0000:a8:00.1 Ethernet controller: Mellanox Technologies MT43244 Family [BlueField-3 integrated ConnectX-7]
0000:a8:00.2 Ethernet controller: Mellanox Technologies MT42822 Family  [BlueField Auxiliary Comm Channel]
0000:a8:00.3 DMA controller: Mellanox Technologies MT43244 BlueField-3 SoC Management Interface

  6. Check the NIC link status from the ESXi service console.

    ESXi console

    [root@qa-esxio-lnv-06:~] esxcli network nic list

    Example output:
    step01.png

  7. Query NVIDIA device firmware.

    ESXi console

    [root@qa-esxio-lnv-06:~] esxcli network nic list

    Example output:
    step03.png

    If the following is printed: 

    /opt/mellanox/bin/mlxfwmanager
-E- No devices found or specified, mst might be stopped, run 'mst start' to load MST modules

    Then the device is in DPU mode, so MFT does not recognize it. 

  8. Verify whether each device is in DPU mode by ensuring that INTERNAL_CPU_OFFLOAD_ENGINE is enabled.

    ESXi console

    [root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 q  | grep -i offload
        INTERNAL_CPU_OFFLOAD_ENGINE                 ENABLED(0)
[root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf1 q  | grep -i offload
        INTERNAL_CPU_OFFLOAD_ENGINE                 ENABLED(0)

  9. Enter device recovery mode and reboot the server.

    ESXi console

    [root@qa-esxio-lnv-06:~] esxcli system module parameters set -m nmlx5_core -p mst_recovery=1
[root@qa-esxio-lnv-06:~] reboot -f

  10. Configure the desired device to NIC mode.

    ESXi console

    [root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 set INTERNAL_CPU_OFFLOAD_ENGINE=1

    Example output
    step05.png

  11. Power cycle the system after switching to NIC mode.

  12. Verify that INTERNAL_CPU_OFFLOAD_ENGINE is DISABLED after power cycling the system.ESXi console

    [root@qa-esxio-lnv-06:~] /opt/mellanox/bin/mlxconfig -d mt41692_pciconf0 q  | grep -i offload

    Example output: step06.png

Option 3 - Move to NIC Mode from the DPU UEFI Menu

We will add a DPU to a VM as a passthrough PCI device and will connect to DPU UEFI to configure DPU in NIC Mode. 

  1. Change the management interfaces to passthrough. NIC mode from UEFI 01.png

  2. Create a virtual machine (VM) and assign the PCIe passthrough devices. NIC mode from UEFI 02.png

  3. Power on the VM.

  4. Deploy Linux OS from PXE.

  5. Install the driver to get RShim and MFT: 

    Installing minicom is necessary.

    • For RedHat, run:

      ESXi console

      yum install minicom
yum install pv
yum install gcc-gfortran

    • For Ubuntu, run:

      ESXi console

      apt install minicom
apt install pv

    • For SLES, use yast or zypper. For example:

      ESXi console

      zypper -i minicom
zypper -i pv

  6. Connect to DPU via minicom: 

    If you have multiple DPUs on your setup, this means that you have one RShim per DPU (e.g., rshim0 for DPU#1,  rshim1 for DPU#2, etc.)

    Open as many threads on VNC to get access to each DPU using the following formula: 

    minicom -D /dev/rshim<N>/console

    Where <N> represents the index associated with each DPU.

    NIC mode from UEFI 03.png

  7. Open a new SSH connection to the same VM and reset each DPU from the host: 

    ESXi console

    echo "SW_RESET 1" > /dev/rshim<N>/misc

  8. Navigate to the console for your DPU and press the ESC button key on the keyboard.

  9. Define a new password for the DPU.

    The default password for the UEFI BIOS is bluefield.

    NIC mode from UEFI 04.png

  10. Select Device Manager.
    NIC mode from UEFI 05.png

  11. Select Network Device List.
    NIC mode from UEFI 06.png

  12. Select the network device that presents the uplink (i.e., select the device with the uplink MAC address).
    NIC mode from UEFI 07.png

  13. Select NVIDIA Network adapter - $<uplink-mac>.
    NIC mode from UEFI 08.png

  14. Select BlueField Internal Cpu Configuration.
    NIC mode from UEFI 09.png

  15. Set NIC mode by setting Internal CPU Offload Engine to Disabled.
    NIC mode from UEFI 10.png


    To set DPU mode, set Internal CPU Offload Engine to Enabled.

  16. Power cycle the system.

Author


BK.jpg

Boris Kovalev

Boris Kovalev has worked for the past several years as a Solutions Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website.




Last updated: