Networking DOCA SDK Documentation

DOCA XPlane Service Guide

This guide describes how to use the NVIDIA DOCA™ XPlane service on hosts with NVIDIA® ConnectX®-8 or later SuperNICs.

Introduction

DOCA XPlane is a background service that manages local and remote plane failover in OVS-DOCA. It consumes route-availability updates from an NVIDIA® Spectrum-X fabric and programs OpenFlow rules for traffic failover. It also exposes gRPC telemetry for monitoring service health and topology state.

Requirements

To use the DOCA XPlane service, your environment must meet the following hardware and firmware prerequisites:

  • Hardware: A single-port ConnectX-8 or later SuperNIC.

  • Firmware version: 40.48.1000 or later.

Host Configuration

Before you deploy the DOCA XPlane service, complete the following procedures.

Configure Firmware Settings on the SuperNIC

You must configure each SuperNIC: enable multi-plane mode, configure the number of planes, and set the number of physical functions (PFs) to match the number of planes.

  1. Run the following mlxconfig commands to configure the SuperNIC:

    Bash
    sudo mlxconfig -d <mst-device> -y set SRIOV_EN=1
    sudo mlxconfig -d <mst-device> -y set MODULE_SPLIT_M0[0..7]=1 MODULE_SPLIT_M0[8..15]=FF
    sudo mlxconfig -d <mst-device> -y set LAG_RESOURCE_ALLOCATION=1
    sudo mlxconfig -d <mst-device> -y set LINK_TYPE_P1=2
    sudo mlxconfig -d <mst-device> -y set LOAD_BALANCE_MODE_P1=2
    sudo mlxconfig -d <mst-device> -y set NUM_OF_PF=4 NUM_OF_VFS=4 NUM_OF_PLANES_P1=4
    
  2. To apply the configuration, gracefully shut down and power cycle the host.

The OVS-DOCA service must be stopped while you configure devlink parameters.

For each SuperNIC that DOCA XPlane manages, complete the following steps:

  1. Get the PCI address of each PF. For example, run lshw:

    root@server:~# lshw -class net -businfo
    Bus info          Device          Class          Description
    ============================================================
    ...
    pci@0000:9f:00.0  enp159s0f0np0   network        CX8 Family [ConnectX-8]
    pci@0000:9f:00.1  enp159s0f1np1   network        CX8 Family [ConnectX-8]
    pci@0000:9f:00.2  enp159s0f2np2   network        CX8 Family [ConnectX-8]
    pci@0000:9f:00.3  enp159s0f3np3   network        CX8 Family [ConnectX-8]
    ...
    

    In steps 2 through 4, replace the example PCI addresses with the addresses from step 1, in the form pci/<address>.

  2. For each PF in that SuperNIC, set flow_steering_mode to hmfs:

    Bash
    sudo devlink dev param set pci/0000:9f:00.0 name flow_steering_mode cmode runtime value hmfs
    sudo devlink dev param set pci/0000:9f:00.1 name flow_steering_mode cmode runtime value hmfs
    sudo devlink dev param set pci/0000:9f:00.2 name flow_steering_mode cmode runtime value hmfs
    sudo devlink dev param set pci/0000:9f:00.3 name flow_steering_mode cmode runtime value hmfs
    
  3. For each PF in that SuperNIC, set the eswitch mode to switchdev:

    Bash
    sudo devlink dev eswitch set pci/0000:9f:00.0 mode switchdev
    sudo devlink dev eswitch set pci/0000:9f:00.1 mode switchdev
    sudo devlink dev eswitch set pci/0000:9f:00.2 mode switchdev
    sudo devlink dev eswitch set pci/0000:9f:00.3 mode switchdev
    
  4. For each PF in that SuperNIC, set esw_multiport to true:

    Bash
    sudo devlink dev param set pci/0000:9f:00.0 name esw_multiport value true cmode runtime
    sudo devlink dev param set pci/0000:9f:00.1 name esw_multiport value true cmode runtime
    sudo devlink dev param set pci/0000:9f:00.2 name esw_multiport value true cmode runtime
    sudo devlink dev param set pci/0000:9f:00.3 name esw_multiport value true cmode runtime
    

Create SR-IOV Virtual Functions

For each SuperNIC that DOCA XPlane manages, create one virtual function (VF) on the first PF of the SuperNIC. In the following command, replace enp159s0f0np0 with the netdev name of that PF:

Bash
echo 1 | sudo tee /sys/class/net/enp159s0f0np0/device/sriov_numvfs > /dev/null

For more information about configuring SR-IOV, see HowTo Configure SR-IOV for ConnectX-4/ConnectX-5/ConnectX-6 with KVM (Ethernet).

Configure Hugepages and OVS-DOCA Settings

  1. Use the doca-hugepages tool to provision at least 150 hugepages of 2048 KB for each SuperNIC that DOCA XPlane manages.

    For example, on a server with 4 ConnectX-8 SuperNICs, run:

    Bash
    sudo doca-hugepages config --app doca-xplane --size 2048 --num 600
    sudo doca-hugepages reload
    
  2. Set the following Open vSwitch (OVS) other_config options:

    Bash
    sudo ovs-vsctl set Open_vSwitch . other_config:doca-init=true
    sudo ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
    sudo ovs-vsctl set Open_vSwitch . other_config:max-idle=300000
    sudo ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
    sudo ovs-vsctl set Open_vSwitch . other_config:pmd-sleep-max=50
    
  3. Restart OVS-DOCA so that it uses the newly provisioned hugepages and settings:

    Bash
    # Debian and derivatives
    sudo systemctl restart openvswitch-switch
    
    # RHEL and derivatives
    sudo systemctl restart openvswitch
    

XPlane Service

To deploy the DOCA XPlane service, follow these steps:

  1. Install the doca-xplane package from DOCA Host, available for download from the NVIDIA Developer website.

  2. After installation, populate the configuration file as described in the Configuration File section of this page.

  3. After you save the configuration file, start the DOCA XPlane service by running the following command:

    Bash
    sudo systemctl start doca-xplane
    

Command-line Client

DOCA XPlane ships with a command-line client, doca-xplane-client, that retrieves topology and observability data through gRPC.

To use the client, install the doca-xplane-client package from DOCA Host, available for download from the NVIDIA Developer website.

Configuration File

Before you start the service, you must populate the configuration file, /opt/mellanox/doca/services/xplane/xplane.toml, with the list of interfaces that the service uses.

The following table describes the settings that you can configure:

Setting Name

Description

Example Value

operational_mode

Omit this setting, or set it to spx-ra.

spx-ra

interfaces.mtu

The MTU applied to all PFs that DOCA XPlane manages.

The MTU that ip link reports is the configured value plus 50 bytes.

9216

interfaces.groups.GROUP_NAME

The group identifier for a set of PFs and an optional VF. Each group maps related interfaces that DOCA XPlane manages together.

[interfaces.groups.rail0]

interfaces.groups.GROUP_NAME.pfs_plane_map

A map of PF netdev names to plane IDs. The PFs are attached to the br-xplane bridge for this group.

{"eth_p0_rail0" = 0, "eth_p1_rail0" = 1, "eth_p2_rail0" = 2, "eth_p3_rail0" = 3}

interfaces.groups.GROUP_NAME.vf_rep

The name of the VF representor interface that the group uses.

"eth_vf0_rail0"

logging.level

The service logging level. Supported values are debug, info, warn, and error.

"info"

Example for GB200

[interfaces]
mtu = 9216

[interfaces.groups.rail0]
pfs_plane_map = {"eth_r0_p0" = 0, "eth_r0_p1" = 1, "eth_r0_p2" = 2, "eth_r0_p3" = 3}
vf_rep = "eth_vf_rep_r0"

[interfaces.groups.rail1]
pfs_plane_map = {"eth_r1_p0" = 0, "eth_r1_p1" = 1, "eth_r1_p2" = 2, "eth_r1_p3" = 3}
vf_rep = "eth_vf_rep_r1"

[interfaces.groups.rail2]
pfs_plane_map = {"eth_r2_p0" = 0, "eth_r2_p1" = 1, "eth_r2_p2" = 2, "eth_r2_p3" = 3}
vf_rep = "eth_vf_rep_r2"

[interfaces.groups.rail3]
pfs_plane_map = {"eth_r3_p0" = 0, "eth_r3_p1" = 1, "eth_r3_p2" = 2, "eth_r3_p3" = 3}
vf_rep = "eth_vf_rep_r3"

Example for VR200

This example describes a system with two 4-plane SuperNICs per GPU. Each PF connects to a different top-of-rack switch.

[interfaces]
mtu = 9216

[interfaces.groups.rail0_nic0]
pfs_plane_map = {"eth_r0_p0" = 0, "eth_r0_p1" = 1, "eth_r0_p2" = 2, "eth_r0_p3" = 3}
vf_rep = "eth_vf0_rep_r0"

[interfaces.groups.rail0_nic1]
pfs_plane_map = {"eth_r0_p4" = 4, "eth_r0_p5" = 5, "eth_r0_p6" = 6, "eth_r0_p7" = 7}
vf_rep = "eth_vf1_rep_r0"

[interfaces.groups.rail1_nic0]
pfs_plane_map = {"eth_r1_p0" = 0, "eth_r1_p1" = 1, "eth_r1_p2" = 2, "eth_r1_p3" = 3}
vf_rep = "eth_vf0_rep_r1"

[interfaces.groups.rail1_nic1]
pfs_plane_map = {"eth_r1_p4" = 4, "eth_r1_p5" = 5, "eth_r1_p6" = 6, "eth_r1_p7" = 7}
vf_rep = "eth_vf1_rep_r1"

[interfaces.groups.rail2_nic0]
pfs_plane_map = {"eth_r2_p0" = 0, "eth_r2_p1" = 1, "eth_r2_p2" = 2, "eth_r2_p3" = 3}
vf_rep = "eth_vf0_rep_r2"

[interfaces.groups.rail2_nic1]
pfs_plane_map = {"eth_r2_p4" = 4, "eth_r2_p5" = 5, "eth_r2_p6" = 6, "eth_r2_p7" = 7}
vf_rep = "eth_vf1_rep_r2"

[interfaces.groups.rail3_nic0]
pfs_plane_map = {"eth_r3_p0" = 0, "eth_r3_p1" = 1, "eth_r3_p2" = 2, "eth_r3_p3" = 3}
vf_rep = "eth_vf0_rep_r3"

[interfaces.groups.rail3_nic1]
pfs_plane_map = {"eth_r3_p4" = 4, "eth_r3_p5" = 5, "eth_r3_p6" = 6, "eth_r3_p7" = 7}
vf_rep = "eth_vf1_rep_r3"

Maintenance Mode

To turn maintenance mode on or off for a PF, use doca-xplane-client. For installation instructions, see the Command-line Client section of this page.

To turn on maintenance mode for a PF, run:

Bash
sudo doca-xplane-client maintenance set-link-maintenance --pf_name <pf> --enabled true

To turn off maintenance mode for a PF, run:

sudo doca-xplane-client maintenance set-link-maintenance --pf_name <pf> --enabled false

To get the maintenance status of a PF, run:

sudo doca-xplane-client maintenance get-link-maintenance --pf_name <pf>

The response is similar to the following:

JSON
{
  "enabled": false
}

Monitoring

Use the following commands and logs to monitor the DOCA XPlane service.

Check System Status

To check the status of the DOCA XPlane service, run the following command:

Bash
sudo systemctl status doca-xplane

The output is similar to the following:

● doca-xplane.service - DOCA XPlane
     Loaded: loaded (/lib/systemd/system/doca-xplane.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2026-04-23 11:40:08 IDT; 12h ago
   Main PID: 71852 (docker)
...

Service Logs

The DOCA XPlane service stores its logs permanently in /var/log/xplane/.

Using doca-xplane-client

The following examples show how to query the DOCA XPlane service with doca-xplane-client. The outputs are trimmed.

Get Overall System Status

To get the overall status of the service, run:

Bash
sudo doca-xplane-client get-status

The response is similar to the following:

JSON
{
  "serviceName": "xplane",
  "version": "0.1.0",
  "process": {
    "state": "RUNNING",
    "uptimeSeconds": "86400"
  },
  "topologySummary": {
    "groups": 2,
    "planes": 2
  },
  "health": {
    "planesUp": [1],
    "planesDegraded": [0],
    "overallState": "DEGRADED",
    "failures": {
      "localTotal": 1,
      "remoteTotal": 2
    }
  }
}

Get Topology as Read by DOCA XPlane

To get the topology as DOCA XPlane reads it, run:

sudo doca-xplane-client get-topology

The response is similar to the following:

JSON
{
  "pfs": [
    {
      "plane": 0,
      "group": "rail0nic0",
      "pfName": "eth2",
      "pciAddress": "0000:08:00.0",
      "speedGbps": 200
    }
  ],
  "vfs": [
    {
      "vfRep": "eth2",
      "group": "rail0"
    }
  ],
  "patchPorts": []
}

Get Summary of Configured Planes

To get a summary of the configured planes, run:

sudo doca-xplane-client get-planes-summary

The response is similar to the following:

JSON
{
  "planes": [
    {
      "plane": 0,
      "state": "DEGRADED",
      "failures": {
        "localTotal": 1,
        "remoteTotal": 2
      },
      "traffic": {
        "packets": "1250000",
        "bytes": "900000000"
      }
    }
  ]
}

Get Local Failures for a Specific Plane

Local failures include link-down events. To get the local failures for a specific plane, run:

sudo doca-xplane-client get-plane-failures-local --plane_id 0

The response is similar to the following:

JSON
{
  "localFailures": [
    {
      "group": "rail0-nic0",
      "pfName": "eth2"
    }
  ]
}

Get Remote Failures for a Specific Plane

To get the remote failures for a specific plane, run:

sudo doca-xplane-client get-plane-failures-remote --plane_id 0

The response is similar to the following:

JSON
{
  "remoteFailures": [
    {
      "cidr": "10.10.0.0/24",
      "groups": []
    },
    {
      "cidr": "2001:db8:10::/64",
      "groups": []
    },
    {
      "cidr": "0.0.0.0/0",
      "groups": [
        "2",
        "3"
      ]
    }
  ]
}

Get Traffic Summary for a Specific Plane

To get the traffic summary for a specific plane, run:

sudo doca-xplane-client get-plane-traffic-summary --plane_id 0

The response is similar to the following:

JSON
{
  "nonDiverted": {
    "packets": "1200000",
    "bytes": "864000000"
  },
  "divertedTo": {
    "packets": "50000",
    "bytes": "36000000"
  },
  "dropped": {
    "packets": "0",
    "bytes": "0"
  }
}

Get Statistics for Traffic Diverted From One Plane to Another

To see failover routing in both directions, query the traffic diverted from and to a specific plane.

To get the traffic diverted from the plane, run:

sudo doca-xplane-client get-plane-traffic-diverted-from --plane_id 0

The response is similar to the following:

JSON
{
  "aggregated": {
    "packets": "75000",
    "bytes": "54000000"
  },
  "perPlane": {
    "1": {
      "packets": "75000",
      "bytes": "54000000",
      "percentage": 100
    }
  }
}

To get the traffic diverted to the plane, run:

sudo doca-xplane-client get-plane-traffic-diverted-to --plane_id 0

The response is similar to the following:

JSON
{
  "aggregated": {
    "packets": "50000",
    "bytes": "36000000"
  },
  "perPlane": {
    "1": {
      "packets": "50000",
      "bytes": "36000000",
      "percentage": 100
    }
  }
}


Last updated: