Networking DPF Performance Optimizations
Networking DPF Performance Optimizations
Breadcrumbs

HBN-OVN Performance Optimizations

This document specifies some recommended Performance Optimization steps for the HBN-OVN DPF Deployment:

Setting 9K MTU on Management Network

OVN-K8s is deployed as the primary Kubernetes network, and it utilizes both the Management Network and the Highspeed Network. Therefore, in order to use 9K MTU on the Highspeed Network and achieve maximum performance, you must also set 9K MTU on the Management Network. Make sure to do so on the ports of your Management Switch and on all the Management Ports of the nodes connected to the Management Network (e.g. Firewall/Router Node, Master Nodes, Node serving the BFB image, etc.)

Kernel Parameters

The following kernel parameters can be added to worker nodes to make sure that the the cores that belong to the same NUMA Node as the DPU are isolated and reserved for workload pods. Please substitute the values for isolcpus, nohz_full, and rcu_nocbs with the CPU cores from the NUMA node which the BlueField-3 is connected to:

intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111

Setting kubelet Parameters

The following parameters can be set in kubelet configuration file (typically at: /var/lib/kubelet/config.yaml) for using the "single-numa-node" Topology Manager Policy in which Kubernetes assigns to workload pods CPU cores only from the NUMA Node that the DPU is connected to.

Add the following lines to the file:

cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node

Adding 9K MTU to OVN-K8s CNI

Add the following line to the OVN-K8s CNI helm values YAML (manifests/01-cni-installation/helm-values/ovn-kubernetes.yml):

mtu: 8940

For example:

commonManifests:
  enabled: true
nodeWithoutDPUManifests:
  enabled: true
controlPlaneManifests:
  enabled: true
nodeWithDPUManifests:
  enabled: true
  nodeMgmtPortNetdev: $DPU_P0_VF1
  dpuServiceAccountNamespace: dpf-operator-system
gatewayOpts: --gateway-interface=$DPU_P0
## Note this CIDR is followed by a trailing /24 which informs OVN Kubernetes on how to split the CIDR per node.
podNetwork: $POD_CIDR/24
serviceNetwork: $SERVICE_CIDR
k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT
mtu: 8940

Adding 9K MTU to OperatorConfig YAML

Add the controlPlaneMTU and the highSpeedMTU parameters to the OperatorConfig YAML, for example:

---
apiVersion: operator.dpu.nvidia.com/v1alpha1
kind: DPFOperatorConfig
metadata:
  name: dpfoperatorconfig
  namespace: dpf-operator-system
spec:
  overrides:
    kubernetesAPIServerVIP: $TARGETCLUSTER_API_SERVER_HOST
    kubernetesAPIServerPort: $TARGETCLUSTER_API_SERVER_PORT
  provisioningController:
    bfbPVCName: "bfb-pvc"
    dmsTimeout: 900
  kamajiClusterManager:
    disable: false
  networking:
    controlPlaneMTU: 9000
    highSpeedMTU: 9000

Adding 9K MTU to the DPUFlavor YAML

Please add the following lines to the OVS commands script section in the DPUFlavor YAML:

_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Interface br-ovn mtu_request=9216
_ovs-vsctl set Interface pf0hpf mtu_request=9216

For example:

_ovs-vsctl set Open_vSwitch . other_config:doca-init=true
_ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
_ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
_ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
_ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
_ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
_ovs-vsctl --if-exists del-br ovsbr1
_ovs-vsctl --if-exists del-br ovsbr2
_ovs-vsctl --may-exist add-br br-sfc
_ovs-vsctl set bridge br-sfc datapath_type=netdev
_ovs-vsctl set bridge br-sfc fail_mode=secure
_ovs-vsctl --may-exist add-port br-sfc p0
_ovs-vsctl set Interface p0 type=dpdk
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Port p0 external_ids:dpf-type=physical
_ovs-vsctl --may-exist add-port br-sfc p1
_ovs-vsctl set Interface p1 type=dpdk
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Port p1 external_ids:dpf-type=physical

_ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev
_ovs-vsctl --may-exist add-br br-ovn
_ovs-vsctl set bridge br-ovn datapath_type=netdev
_ovs-vsctl set Interface br-ovn mtu_request=9216
_ovs-vsctl --may-exist add-port br-ovn pf0hpf
_ovs-vsctl set Interface pf0hpf type=dpdk
_ovs-vsctl set Interface pf0hpf mtu_request=9216

Increasing Hugepages Allocation in the DPUFlavor YAML

Please increase the hugepages allocation in the DPUFlavor YAML:

    - hugepages=8072

Adding 9K MTU to OVN ServiceConfig YAML

Please add the following line to the OVN ServiceConfig YAML (manifests/05-dpudeployment-installation/dpuserviceconfig_ovn.yaml): mtu: 8940

For example:

---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "ovn"
  serviceConfiguration:
    helmChart:
      values:
        k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT
        podNetwork: $POD_CIDR/24
        serviceNetwork: $SERVICE_CIDR
        mtu: 8940
        dpuManifests:
          kubernetesSecretName: "ovn-dpu" # user needs to populate based on DPUServiceCredentialRequest
          vtepCIDR: "10.0.120.0/22" # user needs to populate based on DPUServiceIPAM
          hostCIDR: $TARGETCLUSTER_NODE_CIDR # user needs to populate
          ipamPool: "pool1" # user needs to populate based on DPUServiceIPAM
          ipamPoolType: "cidrpool" # user needs to populate based on DPUServiceIPAM
          ipamVTEPIPIndex: 0
          ipamPFIPIndex: 1

Last updated: