DPUNode

The DPUNode is a Kubernetes CRD that represents a physical host node containing one or more DPU (Data Processing Unit) devices in the DOCA Platform Framework (DPF). It provides node-level management capabilities for DPU provisioning, reboot control, and integration with Kubernetes clusters.

Overview

The DPUNode resource serves as a bridge between physical host nodes and DPU devices, enabling centralized management of DPU provisioning and host operations. It defines how DPUs should be provisioned on a specific node and how the host should be managed during DPU operations.

Key Features

• Node-Level Management: Manages DPU operations at the host node level

• Reboot Control: Configurable host reboot methods (hostAgent [default], external, script; gNOI is deprecated)

• DPU Association: Links multiple DPU devices to a single node

• Kubernetes Integration: Optional integration with Kubernetes Node objects

DPUNode Specification

DPUNodeSpec

The spec section defines the desired configuration for the DPU node:

NodeRebootMethod

Defines how the host should be rebooted during DPU operations. Exactly one of the fields below must be set; the API server defaults nodeRebootMethod to {hostAgent: {}} when omitted.

DMSAddress

Deprecated configuration for Device Management Service communication. This type is retained for API compatibility through the deprecated nodeDMSAddress field, but is no longer used.

DPURef

Reference to a DPU device:

DPUNodeStatus

The status section contains the observed state of the DPU node:

Aggregated reboot method

status.rebootMethod tells the cluster admin -- or the automation acting on their behalf -- which host operation DPF needs before the DPUs can finish provisioning. The operations DPF may ask for are:

Performing a PowerCycle when SystemLevelReset was requested is safe, only more disruptive. The opposite is not. Any other value that may appear in this field should be mapped into the SystemLevelReset operation.

On a multi-DPU host the field carries the most disruptive operation requested by any child DPU in DPURebooting, so PowerCycle wins. The value is kept after the reboot completes and is cleared only once the DPUNode has no DPUs left.

Conditions

The DPUNode resource uses several condition types to track its state:

• Ready: The DPU node is ready for operations

• InvalidDPUDetails: The DPU details provided are invalid

• DPUNodeRebootInProgress: The DPUNode is in the process of rebooting

• DPUUpdateInProgress: The DPU is being updated

• NeedHostAgentUpgrade: The host agent needs to be upgraded

• OOBBridgeConfigured: The out-of-band bridge (br-dpu) is configured

• RshimAvailable: The rshim interface is available

• DPUNodeNodeEffectInProgress: A node effect is being applied on the node

Example Usage

Basic DPUNode with hostAgent Reboot (default)

---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUNode
metadata:
  name: dpu-node-001
  namespace: dpf-operator-system
spec:
  nodeRebootMethod:
    hostAgent: {}
  dpus:
  - name: dpu-device-001
  - name: dpu-device-002

nodeRebootMethod may be omitted; the API server defaults it to {hostAgent: {}}.

DPUNode with External Reboot

---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUNode
metadata:
  name: dpu-node-002
  namespace: dpf-operator-system
spec:
  nodeRebootMethod:
    external: {}
  dpus:
  - name: dpu-device-003

DPUNode with Custom Script Reboot

---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUNode
metadata:
  name: dpu-node-003
  namespace: dpf-operator-system
spec:
  nodeRebootMethod:
    script:
      name: custom-reboot-script
  dpus:
  - name: dpu-device-004

Custom Reboot Script ConfigMap

The pod-template key must contain a Kubernetes PodTemplateSpec in YAML or JSON format. Do not include apiVersion or kind -- the controller wraps the template inside a Job automatically. The controller injects the DPUNODE_NAME environment variable, a dpf-pod-info downward-API volume, and a control-plane toleration (node-role.kubernetes.io/control-plane:NoSchedule). The Job is created with backoffLimit: 3, so Kubernetes automatically retries the pod up to 3 times with exponential backoff before reporting a terminal failure. If the Job still fails after those retries, see Script reboot job failures and recovery.

Reboot context exposed to the script (v26.4)

Starting with DPF v26.4, the controller also propagates the aggregated reboot intent reported by the DPU agent(s) into every container and init container of the script pod, and stamps it as pod-template annotations. Custom reboot scripts can use this signal to choose the right host action without having to read the DPU API directly.

DPUNODE_REBOOT_METHOD is never empty and never Unknown. When no DPU has reported yet, it is set to SystemLevelReset -- the safe middle ground that brings the host OS down without escalating to a full PowerCycle.

DPUNODE_REBOOT_METHOD and DPUNODE_REBOOT_METHODS_PER_DPU reflect the data-plane recommendation from the DPU agent. The provisioning.dpu.nvidia.com/host-power-cycle-required annotation is a Trusted Host (host agent) execution-time escalation and is not propagated into either of these surfaces. Scripts that need to honor the annotation must read it directly from the DPU resource themselves.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-reboot-script
  namespace: dpf-operator-system
data:
  pod-template: |
    spec:
      containers:
      - name: reboot-container
        image: ubuntu:20.04
        command: ["/bin/bash"]
        args:
        - -c
        - |
          echo "Performing custom reboot procedure for DPUNode $DPUNODE_NAME..."
          echo "Aggregated reboot method: $DPUNODE_REBOOT_METHOD"
          echo "Per-DPU methods: $DPUNODE_REBOOT_METHODS_PER_DPU"
          case "$DPUNODE_REBOOT_METHOD" in
            PowerCycle)
              # Bring the host OS, DPU BMC and DPU OS down, wait, then power
              # the server back up (for example via the host BMC).
              ;;
            SystemLevelReset)
              # Bring the host OS down, wait, then bring it back up.
              ;;
            *)
              # Safe fallback for any other value: handle it as SystemLevelReset.
              ;;
          esac
          sleep 10
          exit 0
      restartPolicy: Never

Reboot Methods

hostAgent (Default)

Uses the on-host DPF agent to reboot the host. This is the default and recommended method for most deployments.

Advantages:

• No DMS dependency or out-of-band channel

• Driven by the same agent that already runs on the host

• Sets Status.RebootMethod (it does not set DPUNodeRebootInProgress; the DPU controller drives the reboot)

Requirements:

• Host agent installed and running on the node

External

Reboots the host via external means not controlled by the DPU controller. This method requires manual intervention or external automation. Read status.rebootMethod to learn which host operation is expected -- see Aggregated reboot method.

Use Cases:

• Custom power management systems

• IPMI-based reboots

• Cloud provider APIs

Requirements:

• External reboot mechanism must be available

• Manual intervention may be required

Script

Executes a custom script to reboot the host. The script is defined in a ConfigMap and executed as a Kubernetes Job.

Use Cases:

• Custom reboot procedures

• Integration with existing automation

• Complex reboot workflows

Requirements:

• ConfigMap with pod template

• Script must exit successfully

• Proper RBAC permissions

gNOI (Deprecated)

Retained for API compatibility. This method uses the host-agent reboot flow. Use hostAgent for new deployments.

Integration with Kubernetes

Node Association

DPUNode can optionally be associated with a Kubernetes Node object:

status:
  kubeNodeRef: "worker-node-001"

This association enables:

• Node-level operations (draining, tainting)

• Integration with Kubernetes scheduling

• Resource management alignment

Annotations

For external reboot, the controller adds the following annotation when the host must be rebooted manually:

metadata:
  annotations:
    provisioning.dpu.nvidia.com/dpunode-external-reboot-required: "true"

Perform the operation reported in status.rebootMethod (see Aggregated reboot method) and remove the annotation only after the host is back up.

Lifecycle Management

Creation

DPUNode resources are typically created:

• Manually: By administrators for known nodes

• Automatically: Via discovery processes

• Via host agent/node manager: In host-trusted deployments

Updates

Some DPUNode fields can be updated, but updates during active provisioning are restricted:

• kubeNodeRef is immutable once set

• dpus list can be modified to add/remove devices

• nodeRebootMethod should not be changed while associated DPUs are actively provisioning

Deletion

DPUNode resources are protected by a finalizer (provisioning.dpu.nvidia.com/dpunode-protection) to prevent deletion while DPUs are in use.

Monitoring and Troubleshooting

Checking Node Status

# Get all DPUNode resources
kubectl get dpunodes -n dpf-operator-system

# Get detailed information about a specific node
kubectl describe dpunode dpu-node-001 -n dpf-operator-system

# Check node conditions
kubectl get dpunode dpu-node-001 -n dpf-operator-system -o jsonpath='{.status.conditions}'

Common Issues

• Invalid DMS Address: Verify IP and port configuration

• DPU Not Found: Ensure referenced DPUDevice resources exist

• Reboot Failures: Check reboot method configuration and permissions

• Script Execution Errors: Verify ConfigMap and script syntax. See Script reboot job failures and recovery below.

Script reboot job failures and recovery

The controller creates one Job per DPUNode in the same namespace as the DPUNode. The Job name is <dpunode-name>-script-job (for example, DPUNode dpu-node-003 uses Job dpu-node-003-script-job).

Kubernetes applies backoffLimit (default 3) before the Job is considered to have failed terminally.

Observed behavior on failure

• Affected DPUs stay in phase DPURebooting until the host reboot completes successfully and the Rebooted status condition becomes True.

• After a terminal Job failure, the Rebooted condition is set to False with reason RebootScriptFailed and a message that includes pod or Job failure details (for example, BackoffLimitExceeded).

Recovery

1. Inspect the Job and pods: kubectl describe job <dpunode-name>-script-job -n <namespace> and kubectl logs job/<dpunode-name>-script-job -n <namespace> (or the pod name shown in the Job events).

2. Fix the ConfigMap pod template or cluster permissions as needed.

3. Delete the failed Job. The controller recreates it on the next reconcile:

kubectl delete job <dpunode-name>-script-job -n <namespace>

Status Monitoring

# Check if node is ready
kubectl get dpunode dpu-node-001 -n dpf-operator-system -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

# Check reboot status
kubectl get dpunode dpu-node-001 -n dpf-operator-system -o jsonpath='{.status.rebootInProgress}'

# Check install interface
kubectl get dpunode dpu-node-001 -n dpf-operator-system -o jsonpath='{.status.dpuInstallInterface}'

Related Resources

• DPUDevice - Individual DPU device management

• DPU - DPU provisioning and deployment

• DPUSet - Bulk DPU and node management

• DPUDiscovery - Automatic DPU discovery

Last updated: October 14, 2026