Follow this guide from the source GitHub repo at github.com/NVIDIA/doca-platform and moving to the docs/public/user-guides/zero-trust/use-cases/hbn-snap/README.md for better formatting of the code.
This configuration provides instructions for deploying the NVIDIA DOCA Platform Framework (DPF) on high-performance, bare-metal infrastructure in Zero Trust mode, utilizing DPU BMC and Redfish. It focuses on provisioning NVIDIA® BlueField®-3 DPUs using DPF, installing the HBN DPUService on those DPUs and enabling SNAP Storage on those DPUs.
Prerequisites
This guide should be run by cloning the repo from github.com/NVIDIA/doca-platform and moving to the docs/public/user-guides/zero-trust/use-cases/hbn-snap directory.
The system is set up as described in the prerequisites.
In addition, for this use case, the Top of Rack switch(ToR) must support BGP and EVPN, and should be configured to support unnumbered BGP towards the two ports of the DPU, where HBN will act as peer, and advertise routes over BGP to allow for ECMP from the DPU. The storage traffic between the DPU and the remote storage system goes through a VXLAN interface. Additional information about the required switch configuration can be found in the Technology Preview for DPF Zero Trust (DPF-ZT) with SNAP DPU Service in virtio-fs mode.
This guide includes examples for both SNAP Block (NVMe) and SNAP VirtioFS Storage. Depending on the storage type you want to deploy, you need to ensure that the following prerequisites are met:
SNAP Block (NVMe) Prerequisites
An remote SPDK target should be set up to provide persistent storage for SNAP Block Storage.
The SPDK target should be reachable from the DPUs.
The management interface of the SPDK target should be reachable from the control plane nodes.
The following tools must be installed on the machine where the commands contained in this guide run:
kubectl
helm
envsubst
Installation Guide
This guide assumes that the setup includes only 2 workers with DPUs. If your setup has more than 2 workers, then you will need to set additional variables to enable the rest of the DPUs.
0. Required Variables
The following variables are required by this guide. A sensible default is provided where it makes sense, but many will be specific to the target infrastructure.
Commands in this guide are run in the same directory that contains this readme.
Environment variables file
## IP Address for the Kubernetes API server of the target cluster on which DPF is installed.
## This should never include a scheme or a port.
## e.g. 10.10.10.10
export TARGETCLUSTER_API_SERVER_HOST=
## Port for the Kubernetes API server of the target cluster on which DPF is installed.
## e.g. 6443
export TARGETCLUSTER_API_SERVER_PORT=
## Virtual IP used by the load balancer for the DPU Cluster. Must be a reserved IP from the management subnet and not
## allocated by DHCP.
export DPUCLUSTER_VIP=
## Interface on which the DPUCluster load balancer will listen. Should be the management interface of the control plane node.
export DPUCLUSTER_INTERFACE=
## The repository URL for the NVIDIA Helm chart registry.
## Usually this is the NVIDIA Helm NGC registry. For development purposes, this can be set to a different repository.
export HELM_REGISTRY_REPO_URL=https://helm.ngc.nvidia.com/nvidia/doca
## The repository URL for the HBN container image.
## Usually this is the NVIDIA NGC registry. For development purposes, this can be set to a different repository.
export HBN_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_hbn
## The repository URL for the SNAP VFS container image.
## Usually this is the NVIDIA NGC registry. For development purposes, this can be set to a different repository.
export SNAP_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_vfs
## The DPF REGISTRY is the Helm repository URL where the DPF Operator Chart resides.
## Usually this is the NVIDIA Helm NGC registry. For development purposes, this can be set to a different repository.
export REGISTRY=https://helm.ngc.nvidia.com/nvidia/doca
## The DPF TAG is the version of the DPF components which will be deployed in this guide.
export TAG=v26.8.0
## URL to the BFB used in the `bfb.yaml` and linked by the DPUSet.
export BFB_URL="https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod.bfb"
## IP_RANGE_START and IP_RANGE_END
## These define the IP range for DPU discovery via Redfish/BMC interfaces
## Example: If your DPUs have BMC IPs in range 192.168.1.100-110
## export IP_RANGE_START=192.168.1.100
## export IP_RANGE_END=192.168.1.110
export IP_RANGE_START=
export IP_RANGE_END=
# The password used for DPU BMC root login, must be the same for all DPUs
export BMC_ROOT_PASSWORD=
## Serial number of DPUs. If you have more than 2 DPUs, you will need to parameterize the system accordingly and expose
## additional variables.
## All serial numbers must be in lowercase.
export DPU1_SERIAL=
export DPU2_SERIAL=
Modify the variables in manifests/00-env-vars/envvars.env to fit your environment, then source the file:
source manifests/00-env-vars/envvars.env
1. DPF Operator Installation
Create DPU BMC shared password secret
In Zero Trust mode, provisioning DPUs requires authentication with Redfish. In order to do that, you must set the same root password to access the BMC for all DPUs DPF is going to manage.
This shared password is used to authenticate with every DPU BMC discovered by DPUDiscovery. To assign a unique BMC password to individual DPUs instead, see Per-DPU BMC Credentials.
[!NOTE] For production deployments, consider managing this Secret from an external secret store with External Secrets Operator. See Encrypted Secrets for an example.
Additional Dependencies
Before deploying the DPF Operator, ensure that Helm is properly configured according to the Helm prerequisites.
This is a critical prerequisite step that must be completed for the DPF Operator to function properly.
These verification commands may need to be run multiple times to ensure the condition is met.
Verify the DPF Operator installation with:
## Ensure the DPF Operator deployment is available.
kubectl rollout status deployment --namespace dpf-operator-system dpf-operator-controller-manager
## Ensure all pods in the DPF Operator system are ready.
kubectl wait --for=condition=ready --namespace dpf-operator-system pods --all
2. DPF System Installation
This section involves creating the DPF system components and some basic infrastructure required for a functioning DPF-enabled cluster.
DPUCluster to serve as Kubernetes control plane for DPU nodes
YAML
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUCluster
metadata:
name: dpu-cplane-tenant1
namespace: dpu-cplane-tenant1
spec:
type: kamaji
maxNodes: 1000
clusterEndpoint:
# deploy keepalived instances on the nodes that match the given nodeSelector.
keepalived:
# interface on which keepalived will listen. Should be the oob interface of the control plane node.
interface: $DPUCLUSTER_INTERFACE
# Virtual IP reserved for the DPU Cluster load balancer. Must not be allocatable by DHCP.
vip: $DPUCLUSTER_VIP
# virtualRouterID must be in range [1,255], make sure the given virtualRouterID does not duplicate with any existing keepalived process running on the host
virtualRouterID: 126
nodeSelector:
node-role.kubernetes.io/control-plane: ""
These verification commands may need to be run multiple times to ensure the condition is met.
Verify the DPF System with:
## Ensure the provisioning and DPUService controller manager deployments are available.
kubectl rollout status deployment --namespace dpf-operator-system dpf-provisioning-controller-manager dpuservice-controller-manager
## Ensure all other deployments in the DPF Operator system are Available.
kubectl rollout status deployment --namespace dpf-operator-system
## Ensure bfb-registry pod is running.
kubectl wait --for=condition=ready --namespace dpf-operator-system pod/bfb-registry --timeout=600s
## Ensure bfb-registry service exists.
kubectl get svc bfb-registry --namespace dpf-operator-system
## Ensure the DPUCluster is ready for nodes to join.
kubectl wait --for=condition=ready --namespace dpu-cplane-tenant1 dpucluster --all
3. DPU Provisioning and Service Installation
In this section, you will provision your DPUs and deploy the required services. You will need to create a DPUDeployment object that defines which DPUServices should be installed on each selected DPU. This provides a flexible way to specify and manage the services that run on your DPUs.
This guide includes examples for both SNAP Block (NVMe) and SNAP VirtioFS Storage. Please refer to the relevant sections below and follow the instructions to deploy the desired storage type.
Storage use-cases set RDMA_SET_NETNS_EXCLUSIVE="no" in the DPUFlavor, putting the DPU in shared RDMA mode. The default SFC NAD (mybrsfc) enables RDMA for SF interfaces, which is not compatible with shared RDMA mode. All services deployed on a DPU provisioned with a storage flavor that use SF interfaces must reference a NAD without RDMA. A custom DPUServiceNAD (mybrsfc-storage) is included in the manifests below for this reason.
host:
enabled: true
config:
targets:
nodes:
# name of the target
- name: spdk-target
# management address
rpcURL: http://10.0.110.25:8000
# type of the target, e.g. nvme-tcp, nvme-rdma
targetType: nvme-rdma
# target service IP
targetAddr: 10.0.124.1
# required parameter, name of the secret that contains connection
# details to access the DPU cluster.
# this secret should be created by the DPUServiceCredentialRequest API.
dpuClusterSecret: spdk-csi-controller-dpu-cluster-credentials
Apply DPU-side Storage Resources
In case more than 1 DPU exists per node, the relevant selector should be applied in the DPUDeployment to select the appropriate DPU. See DPUDeployment - DPUs Configuration to understand more about the selectors.
HBN DPUFlavor to correctly configure the DPUs on provisioning
YAML
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
name: hbn-snap-nvme-$TAG
namespace: dpf-operator-system
spec:
bfcfgParameters:
- UPDATE_ATF_UEFI=yes
- UPDATE_DPU_OS=yes
- WITH_NIC_FW_UPDATE=yes
configFiles:
- operation: override
path: /etc/mellanox/mlnx-bf.conf
permissions: "0644"
raw: |
ALLOW_SHARED_RQ="no"
IPSEC_FULL_OFFLOAD="no"
ENABLE_ESWITCH_MULTIPORT="yes"
RDMA_SET_NETNS_EXCLUSIVE="no"
- operation: override
path: /etc/mellanox/mlnx-ovs.conf
permissions: "0644"
raw: |
CREATE_OVS_BRIDGES="no"
OVS_DOCA="yes"
- operation: override
path: /etc/mellanox/mlnx-sf.conf
permissions: "0644"
raw: ""
- operation: override
path: /etc/apparmor.d/local/usr.sbin.rsyslogd
permissions: "0644"
raw: |
signal (receive) peer=runc,
capability chown,
/usr/{bin,sbin}/* ixr,
/etc/logrotate.d/* r,
/var/lib/logrotate/{,**} rwk,
- operation: override
path: /etc/apparmor.d/local/usr.bin.tcpdump
permissions: "0644"
raw: |
signal (receive) peer=runc,
grub:
kernelParameters:
- console=hvc0
- console=ttyAMA0
- earlycon=pl011,0x13010000
- net.ifnames=0
- biosdevname=0
- iommu.passthrough=1
- cgroup_no_v1=net_prio,net_cls
- hugepagesz=2048kB
- hugepages=2298
nvconfig:
- device: '*'
parameters:
- PF_BAR2_ENABLE=0
- PER_PF_NUM_SF=1
- PF_TOTAL_SF=20
- PF_SF_BAR_SIZE=10
- NUM_PF_MSIX_VALID=0
- PF_NUM_PF_MSIX_VALID=1
- PF_NUM_PF_MSIX=228
- INTERNAL_CPU_MODEL=1
- INTERNAL_CPU_OFFLOAD_ENGINE=0
- SRIOV_EN=1
- NUM_OF_VFS=46
- LAG_RESOURCE_ALLOCATION=1
- NVME_EMULATION_ENABLE=1
- NVME_EMULATION_NUM_PF=1
- PCI_SWITCH_EMULATION_ENABLE=1
- PCI_SWITCH_EMULATION_NUM_PORT=32
- LINK_TYPE_P1=ETH
- LINK_TYPE_P2=ETH
ovs:
rawConfigScript: |
_ovs-vsctl() {
ovs-vsctl --timeout 15 "$@"
}
# Remove default OVS configuration on the DPU and ensure no leftovers on the OVS kernel side
_ovs-vsctl --if-exists del-br ovsbr1
_ovs-vsctl --if-exists del-br ovsbr2
ovs-appctl --timeout 15 dpctl/del-dp system@ovs-system || true
_ovs-vsctl set Open_vSwitch . other_config:doca-init=true
_ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
_ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
_ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
_ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
_ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
_ovs-vsctl remove Open_vSwitch . other_config default-datapath-type || true
if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
systemctl restart openvswitch-switch
elif systemctl list-unit-files openvswitch.service &>/dev/null; then
systemctl restart openvswitch
fi
_ovs-vsctl --may-exist add-br br-sfc
_ovs-vsctl set bridge br-sfc datapath_type=netdev
_ovs-vsctl set bridge br-sfc fail_mode=secure
_ovs-vsctl --may-exist add-port br-sfc p0
_ovs-vsctl set Interface p0 type=dpdk
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Port p0 external_ids:dpf-type=physical
_ovs-vsctl set Interface p0 external_ids:hbn_uplink=p0_if
_ovs-vsctl --may-exist add-port br-sfc p1
_ovs-vsctl set Interface p1 type=dpdk
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Port p1 external_ids:dpf-type=physical
_ovs-vsctl set Interface p1 external_ids:hbn_uplink=p1_if
_ovs-vsctl --if-exists del-br br-hbn
_ovs-vsctl --may-exist add-br br-hbn
_ovs-vsctl set bridge br-hbn datapath_type=netdev
_ovs-vsctl set bridge br-hbn fail_mode=secure
The DPUFlavor above does not set ubuntu_PASSWORD in bfcfgParameters. If omitted, DPF defaults to ubuntu:ubuntu credentials on the DPU. For production deployments, it is recommended to set a hashed password (see BFCfg Parameters).
DPUServiceConfiguration and DPUServiceTemplate for SPDK CSI Controller on DPU
YAML
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
name: spdk-csi-controller-dpu
namespace: dpf-operator-system
spec:
deploymentServiceName: spdk-csi-controller-dpu
upgradePolicy:
applyNodeEffect: false
serviceConfiguration:
helmChart:
values:
dpu:
enabled: true
storageClass:
# the name of the storage class that will be created for spdk-csi,
# this StorageClass name should be used in the StorageVendor settings
name: spdkcsi-sc
# name of the secret that contains credentials for the remote SPDK target,
# content of the secret is injected during CreateVolume request
secretName: spdkcsi-secret
# namespace of the secret with credentials for the remote SPDK target
secretNamespace: dpf-operator-system
rbacRoles:
spdkCsiController:
# the name of the service account for spdk-csi-controller
# this value must be aligned with the value from the DPUServiceCredentialRequest
serviceAccount: spdk-csi-controller-sa
The Secret below contains credentials. For production deployments, consider managing this Secret from an external secret store with External Secrets Operator. See Encrypted Secrets for an example.
YAML
---
apiVersion: v1
kind: Secret
metadata:
name: spdkcsi-secret
namespace: dpf-operator-system
labels:
# this label enables replication of the secret from the host to the dpu cluster
dpu.nvidia.com/image-pull-secret: ""
stringData:
# name field in the "rpcTokens" list should match name of the
# spdk target from DPUService.helmChart.values.host.config.targets.nodes
secret.json: |-
{
"rpcTokens": [
{
"name": "spdk-target",
"username": "exampleuser",
"password": "examplepassword"
}
]
}
Verification
These verification commands may need to be run multiple times to ensure the condition is met.
Note that the DPUService name will have a random suffix. For example, doca-hbn-l2xsl.
Verify the DPU and Service installation with:
## Ensure the BFB is ready
kubectl wait --for=jsonpath='{.status.phase}'=Ready --namespace dpf-operator-system bfb bf-bundle-$TAG --timeout=600s
## Ensure the DPUServices are created and have been reconciled.
kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_hbn-snap
## Ensure the DPUServiceIPAMs have been reconciled
kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
## Ensure the DPUServiceInterfaces have been reconciled
kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
## Ensure the DPUServiceChains have been reconciled
kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
## Ensure the DPUs have the condition Initialized (this may take time)
kubectl wait --for=condition=Initialized --namespace dpf-operator-system dpu --all
Be sure to create DPUVolumeAttachments that are using static PFs before rebooting the hosts. If you reboot the hosts without creating these attachments, the hardware initialization process on the host can be significantly delayed, due to timeouts caused by partially initialized emulated NVMe controllers. Creating the attachments ahead of time ensures that SNAP services can complete initialization of the emulated NVMe controllers right after the DPUs become ready.
Releasing the Node Effect Hold
Since the DPUDeployment is configured with nodeEffect.hold: true, the DPUs will pause at the "Node Effect" phase and wait for external action before proceeding with provisioning. This gives the administrator control over when the node effect is applied.
To check that DPUNodeMaintenance objects have been created and are in the hold state:
kubectl get dpunodemaintenances -n dpf-operator-system
Once you are ready for provisioning to proceed, release the hold by setting the annotation on the DPUNodeMaintenance objects to "false". You can do this per-node or all at once:
After releasing the hold, the DPUs will proceed through the remaining provisioning phases (BFB installation, OS installation, etc.).
Making the DPUs Ready
In order to make the DPUs ready, the host operation that DPF requests in DPUNode.status.rebootMethod must be performed -- see DPUNode: Aggregated reboot method for what each value requires. With nodeRebootMethod: external we perform it manually when the object gives us the signal; the described flow can be automated by the admin depending on the infrastructure.
The following verification command may need to be run multiple times to ensure the condition is met.
## Ensure the DPUs are in the Rebooting phase with rebootStatus.reason=WaitingForManualPowerCycleOrReboot
kubectl wait --for=jsonpath='{.status.rebootStatus.reason}'=WaitingForManualPowerCycleOrReboot --namespace dpf-operator-system dpu --all
## Check which operation each host needs (see DPUNode: Aggregated reboot method)
kubectl get dpunode --namespace dpf-operator-system -o custom-columns='NAME:.metadata.name,REBOOT_METHOD:.status.rebootMethod'
At this point, we have to perform that operation on the hosts.
For the SNAP Block (NVMe) scenario, you do not need to wait for the host to fully boot after a power cycle before removing the annotation below. On some server platforms, device initialization may be waiting for a long time until the SNAP service on the DPU becomes ready, so it is recommended to remove the annotation immediately after initiating the host power cycle.
Connect to each worker node with a DPU and set the function VUID variable to the value retrieved for that worker.
func_vuid=<set to the function VUID that you retrieved in the previous step>
Discover the function's PCI address from its VPD data in sysfs:
if [ -z "$func_vuid" ]; then
echo "The function VUID is empty. Stop and verify the DPUVolumeAttachment status." >&2
exit 1
fi
pci_address=""
match_count=0
for pci_device in /sys/bus/pci/devices/*; do
if [ -e "$pci_device/physfn" ] || [ ! -r "$pci_device/vpd" ]; then
continue
fi
if grep -F -q -e "$func_vuid" "$pci_device/vpd"; then
pci_address="${pci_device##*/}"
match_count=$((match_count + 1))
fi
done
if [ "$match_count" -ne 1 ]; then
echo "Expected exactly one PCI device for function VUID $func_vuid, found $match_count." >&2
exit 1
fi
echo "NVMe PCI address: $pci_address"
Check the current driver for the device
ls -lah /sys/bus/pci/devices/"$pci_address"/driver
If the device has no driver, you may need to manually bind it.
Find the block device name from its sysfs namespace entry. When native NVMe multipath is enabled, translate the hidden per-controller path name to its usable namespace head name.
block_dev_name=""
match_count=0
for namespace_path in /sys/bus/pci/drivers/nvme/"$pci_address"/nvme/*/nvme*n*; do
if [ ! -r "$namespace_path/nsid" ]; then
continue
fi
candidate="${namespace_path##*/}"
if [[ "$candidate" =~ ^(nvme[0-9]+)c[0-9]+n([0-9]+)$ ]]; then
candidate="${BASH_REMATCH[1]}n${BASH_REMATCH[2]}"
fi
if [ -b "/dev/$candidate" ]; then
block_dev_name="$candidate"
match_count=$((match_count + 1))
fi
done
if [ "$match_count" -ne 1 ]; then
echo "Expected exactly one block device for PCI device $pci_address, found $match_count." >&2
exit 1
fi
echo "Block device name: $block_dev_name"
Format the device to the filesystem or use it as a raw block device.
Example how to format the device with ext4 filesystem and perform I/O operations:
host:
enabled: true
config:
# required parameter, name of the secret that contains connection
# details to access the DPU cluster.
# this secret should be created by the DPUServiceCredentialRequest API.
dpuClusterSecret: nfs-csi-controller-dpu-cluster-credentials
Apply DPU-side Storage Resources
In case more than 1 DPU exists per node, the relevant selector should be applied in the DPUDeployment to select the appropriate DPU. See DPUDeployment - DPUs Configuration to understand more about the selectors.
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
name: hbn-snap-virtiofs-$TAG
namespace: dpf-operator-system
spec:
bfcfgParameters:
- UPDATE_ATF_UEFI=yes
- UPDATE_DPU_OS=yes
- WITH_NIC_FW_UPDATE=yes
configFiles:
- operation: override
path: /etc/mellanox/mlnx-bf.conf
permissions: "0644"
raw: |
ALLOW_SHARED_RQ="no"
IPSEC_FULL_OFFLOAD="no"
ENABLE_ESWITCH_MULTIPORT="yes"
RDMA_SET_NETNS_EXCLUSIVE="no"
- operation: override
path: /etc/mellanox/mlnx-ovs.conf
permissions: "0644"
raw: |
CREATE_OVS_BRIDGES="no"
OVS_DOCA="yes"
- operation: override
path: /etc/mellanox/mlnx-sf.conf
permissions: "0644"
raw: ""
- operation: override
path: /etc/apparmor.d/local/usr.sbin.rsyslogd
permissions: "0644"
raw: |
signal (receive) peer=runc,
capability chown,
/usr/{bin,sbin}/* ixr,
/etc/logrotate.d/* r,
/var/lib/logrotate/{,**} rwk,
- operation: override
path: /etc/apparmor.d/local/usr.bin.tcpdump
permissions: "0644"
raw: |
signal (receive) peer=runc,
grub:
kernelParameters:
- console=hvc0
- console=ttyAMA0
- earlycon=pl011,0x13010000
- net.ifnames=0
- biosdevname=0
- iommu.passthrough=1
- cgroup_no_v1=net_prio,net_cls
- hugepagesz=2048kB
- hugepages=2298
nvconfig:
- device: '*'
parameters:
- PF_BAR2_ENABLE=0
- PER_PF_NUM_SF=1
- PF_TOTAL_SF=20
- PF_SF_BAR_SIZE=10
- NUM_PF_MSIX_VALID=0
- PF_NUM_PF_MSIX_VALID=1
- PF_NUM_PF_MSIX=228
- INTERNAL_CPU_MODEL=1
- INTERNAL_CPU_OFFLOAD_ENGINE=0
- SRIOV_EN=1
- NUM_OF_VFS=46
- LAG_RESOURCE_ALLOCATION=1
- PCI_SWITCH_EMULATION_ENABLE=1
- PCI_SWITCH_EMULATION_NUM_PORT=32
- VIRTIO_FS_EMULATION_ENABLE=1
- VIRTIO_FS_EMULATION_NUM_PF=0
- LINK_TYPE_P1=ETH
- LINK_TYPE_P2=ETH
ovs:
rawConfigScript: |
_ovs-vsctl() {
ovs-vsctl --timeout 15 "$@"
}
# Remove default OVS configuration on the DPU and ensure no leftovers on the OVS kernel side
_ovs-vsctl --if-exists del-br ovsbr1
_ovs-vsctl --if-exists del-br ovsbr2
ovs-appctl --timeout 15 dpctl/del-dp system@ovs-system || true
_ovs-vsctl set Open_vSwitch . other_config:doca-init=true
_ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
_ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
_ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
_ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
_ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
_ovs-vsctl remove Open_vSwitch . other_config default-datapath-type || true
if systemctl list-unit-files openvswitch-switch.service &>/dev/null; then
systemctl restart openvswitch-switch
elif systemctl list-unit-files openvswitch.service &>/dev/null; then
systemctl restart openvswitch
fi
_ovs-vsctl --may-exist add-br br-sfc
_ovs-vsctl set bridge br-sfc datapath_type=netdev
_ovs-vsctl set bridge br-sfc fail_mode=secure
_ovs-vsctl --may-exist add-port br-sfc p0
_ovs-vsctl set Interface p0 type=dpdk
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Port p0 external_ids:dpf-type=physical
_ovs-vsctl set Interface p0 external_ids:hbn_uplink=p0_if
_ovs-vsctl --may-exist add-port br-sfc p1
_ovs-vsctl set Interface p1 type=dpdk
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Port p1 external_ids:dpf-type=physical
_ovs-vsctl set Interface p1 external_ids:hbn_uplink=p1_if
_ovs-vsctl --if-exists del-br br-hbn
_ovs-vsctl --may-exist add-br br-hbn
_ovs-vsctl set bridge br-hbn datapath_type=netdev
_ovs-vsctl set bridge br-hbn fail_mode=secure
The DPUFlavor above does not set ubuntu_PASSWORD in bfcfgParameters. If omitted, DPF defaults to ubuntu:ubuntu credentials on the DPU. For production deployments, it is recommended to set a hashed password (see BFCfg Parameters).
DPUServiceConfiguration and DPUServiceTemplate for NFS CSI Controller on DPU
YAML
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
name: nfs-csi-controller-dpu
namespace: dpf-operator-system
spec:
deploymentServiceName: nfs-csi-controller-dpu
upgradePolicy:
applyNodeEffect: false
serviceConfiguration:
helmChart:
values:
dpu:
enabled: true
storageClasses:
# List of storage classes to be created for nfs-csi
# These StorageClass names should be used in the StorageVendor settings
- name: nfs-csi
parameters:
server: 10.0.124.1
share: /srv/nfs/share
rbacRoles:
nfsCsiController:
# the name of the service account for nfs-csi-controller
# this value must be aligned with the value from the DPUServiceCredentialRequest
serviceAccount: nfs-csi-controller-sa
These verification commands may need to be run multiple times to ensure the condition is met.
Note that the DPUService name will have a random suffix. For example, doca-hbn-l2xsl.
Verify the DPU and Service installation with:
## Ensure the BFB is ready
kubectl wait --for=jsonpath='{.status.phase}'=Ready --namespace dpf-operator-system bfb bf-bundle-$TAG --timeout=600s
## Ensure the DPUServices are created and have been reconciled.
kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_hbn-snap
## Ensure the DPUServiceIPAMs have been reconciled
kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
## Ensure the DPUServiceInterfaces have been reconciled
kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
## Ensure the DPUServiceChains have been reconciled
kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
## Ensure the DPUs have the condition Initialized (this may take time)
kubectl wait --for=condition=Initialized --namespace dpf-operator-system dpu --all
Since the DPUDeployment is configured with nodeEffect.hold: true, the DPUs will pause at the "Node Effect" phase and wait for external action before proceeding with provisioning. This gives the administrator control over when the node effect is applied.
To check that DPUNodeMaintenance objects have been created and are in the hold state:
kubectl get dpunodemaintenances -n dpf-operator-system
Once you are ready for provisioning to proceed, release the hold by setting the annotation on the DPUNodeMaintenance objects to "false". You can do this per-node or all at once:
After releasing the hold, the DPUs will proceed through the remaining provisioning phases (BFB installation, OS installation, etc.).
Making the DPUs Ready
In order to make the DPUs ready, the host operation that DPF requests in DPUNode.status.rebootMethod must be performed -- see DPUNode: Aggregated reboot method for what each value requires. With nodeRebootMethod: external we perform it manually when the object gives us the signal; the described flow can be automated by the admin depending on the infrastructure.
The following verification command may need to be run multiple times to ensure the condition is met.
## Ensure the DPUs are in the Rebooting phase with rebootStatus.reason=WaitingForManualPowerCycleOrReboot
kubectl wait --for=jsonpath='{.status.rebootStatus.reason}'=WaitingForManualPowerCycleOrReboot --namespace dpf-operator-system dpu --all
## Check which operation each host needs (see DPUNode: Aggregated reboot method)
kubectl get dpunode --namespace dpf-operator-system -o custom-columns='NAME:.metadata.name,REBOOT_METHOD:.status.rebootMethod'
At this point, we have to perform that operation on the hosts.
Both Block and VirtioFS scenarios can be tested with the same steps.
After the DPUs are provisioned and the rest of the objects are Ready, we can test traffic by assigning an IP to the PF0 on the host for each DPU, and run a simple ping. Although the configuration is enabling both PFs, we focus on the PF0 for testing traffic. Assuming the PF0 is named ens5f0np0:
On the host with DPU with serial number DPU1_SERIAL:
ip link set dev ens5f0np0 up
ip addr add 10.0.121.1/29 dev ens5f0np0
ip route add 10.0.121.0/24 dev ens5f0np0 via 10.0.121.2
On the host with DPU with serial number DPU2_SERIAL:
ip link set dev ens5f0np0 up
ip addr add 10.0.121.9/29 dev ens5f0np0
ip route add 10.0.121.0/24 dev ens5f0np0 via 10.0.121.10
On the host with DPU with serial number DPU1_SERIAL:
$ ping 10.0.121.9 -c3
PING 10.0.121.9 (10.0.121.9) 56(84) bytes of data.
64 bytes from 10.0.121.9: icmp_seq=1 ttl=64 time=0.387 ms
64 bytes from 10.0.121.9: icmp_seq=2 ttl=64 time=0.344 ms
64 bytes from 10.0.121.9: icmp_seq=3 ttl=64 time=0.396 ms
--- 10.0.121.9 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2053ms
rtt min/avg/max/mdev = 0.344/0.375/0.396/0.022 ms
Uninstall
This section covers only the DPF related components and not the prerequisites as these must be managed by the admin.
Delete storage resources
Be sure to unmount all volumes from the worker nodes before deleting the DPUVolumeAttachments or the operation may fail. For NVMe attachments, it is recommended to unbind the device from the driver before deleting the DPUVolumeAttachment. echo <pci_address> > /sys/bus/pci/drivers/nvme/unbind
Note: there can be a race condition with deleting the underlying Kamaji cluster which runs the DPU cluster control plane in this guide. If that happens it may be necessary to remove finalizers manually from DPUCluster and Datastore objects.
