DOCA XPlane Service Guide

This guide provides instructions on how to use the DOCA XPlane Service on top of hosts equipped with NVIDIA® ConnectX-8 SuperNICs. 

1. Introduction

DOCA XPlane is a background daemon designed to manage local and remote plane failover within OVS DOCA. It actively consumes route-availability updates from an NVIDIA® Spectrum-X fabric, dynamically programs OpenFlow rules to facilitate traffic failover, and exposes comprehensive gRPC telemetry for monitoring service health and topology state.

2. Requirements

To utilize the DOCA XPlane service, your environment must meet the following hardware and firmware prerequisites:

• Hardware: A single-port NVIDIA® ConnectX®-8 SuperNIC.

• Firmware version: Version 40.48.1000 or higher.

2.1. Configuring Firmware Settings on the SuperNIC

Before deploying the service, you must enable multi-plane mode, configure the number of planes, and set the number of Physical Functions (PFs) to match that number on each SuperNIC.

1. Execute the following mlxconfig commands to configure the device.

   Bash

   mlxconfig -d <mst-device> -y set SRIOV_EN=1
   mlxconfig -d <mst-device> -y set MODULE_SPLIT_M0[0..7]=1 MODULE_SPLIT_M0[8..15]=FF
   mlxconfig -d <mst-device> -y set LAG_RESOURCE_ALLOCATION=1
   mlxconfig -d <mst-device> -y set LINK_TYPE_P1=2
   mlxconfig -d <mst-device> -y set LOAD_BALANCE_MODE_P1=2
   mlxconfig -d <mst-device> -y set NUM_OF_PF=4 NUM_OF_VFS=4 NUM_OF_PLANES_P1=4
   

2. Apply the configuration by performing a graceful shutdown and power cycle of the host.

2.2. Setting devlink Parameters

For each SuperNIC managed by DOCA XPlane:

1. Get PCI address of each PF. One easy way to do that is by using lshw:

   root@server:~# lshw -class net -businfo
   Bus info          Device          Class          Description
   ============================================================
   ...
   pci@0000:9f:00.0  enp159s0f0np0   network        CX8 Family [ConnectX-8]
   pci@0000:9f:00.1  enp159s0f1np1   network        CX8 Family [ConnectX-8]
   pci@0000:9f:00.2  enp159s0f2np2   network        CX8 Family [ConnectX-8]
   pci@0000:9f:00.3  enp159s0f3np3   network        CX8 Family [ConnectX-8]
   ...
   

2. For each PF in that SuperNIC, set flow_steering_mode to hmfs:

   Bash

   devlink dev param set pci/0000:9f:00.0 name flow_steering_mode cmode runtime value hmfs
   devlink dev param set pci/0000:9f:00.1 name flow_steering_mode cmode runtime value hmfs
   devlink dev param set pci/0000:9f:00.2 name flow_steering_mode cmode runtime value hmfs
   devlink dev param set pci/0000:9f:00.3 name flow_steering_mode cmode runtime value hmfs
   

3. For each PF in that SuperNIC, set eswitch mode to switchdev:

   Bash

   devlink dev eswitch set pci/0000:9f:00.0 mode switchdev
   devlink dev eswitch set pci/0000:9f:00.1 mode switchdev
   devlink dev eswitch set pci/0000:9f:00.2 mode switchdev
   devlink dev eswitch set pci/0000:9f:00.3 mode switchdev
   

4. For each PF of that SuperNIC, set esw_multiport to true:

   Bash

   sudo devlink dev param set pci/0000:9f:00.0 name esw_multiport value true cmode runtime
   sudo devlink dev param set pci/0000:9f:00.1 name esw_multiport value true cmode runtime
   sudo devlink dev param set pci/0000:9f:00.2 name esw_multiport value true cmode runtime
   sudo devlink dev param set pci/0000:9f:00.3 name esw_multiport value true cmode runtime
   

2.3. Creating SR-IOV Virtual Functions

For each SuperNIC managed by DOCA XPlane, create one VF on the first PF of the SuperNIC.

echo 1 > /sys/class/net/enp159s0f0np0/device/sriov_numvfs

For configuration details, refer to the community guide HowTo Configure SR-IOV for ConnectX-4/ConnectX- 5/ConnectX-6 with KVM (Ethernet).

2.4. Configuring hugepages

1. Use the doca-hugepages tool to provision at least 150 hugepages of 2048KB per SuperNIC managed by DOCA XPlane.
   For instance, for a server equipped with 4 NVIDIA® ConnectX-8 SuperNICs, run:

   Bash

   doca-hugepages config --app doca-xplane --size 2048 --num 600
   doca-hugepages reload
   

2. Restart OVS-DOCA for it to use the newly provisioned hugepages:

   Bash

   # Debian and derivatives
   systemctl restart openvswitch-switch
   
   # RHEL and derivatives
   systemctl restart openvswitch
   

3. Service Deployment

To deploy the DOCA XPlane service, follow these steps:

1. Locate and download the relevant DOCA installation files for your host from the NVIDIA DOCA Downloads page. Install the doca-xplane package.

2. Once installed, populate the configuration file as detailed in the "Configuration File" section.

3. After saving your configuration, start the XPlane daemon by running the following command: 

   Bash

   systemctl start doca-xplane
   

4. Configuration File

Prior to starting the service, you must populate the configuration file located at /opt/mellanox/doca/services/xplane/xplane.toml with the list of interfaces to be used.

The table below details the user-configurable settings and their purposes:

true

true

true

9216

"rail0"

["eth_p0_rail0", "eth_p1_rail0", "eth_p2_rail0", "eth_p3_rail0"] 

"eth_vf0_rail0"

"info"

5. Troubleshooting

5.1. Checking System Status

Verify the current operational status of the DOCA XPlane daemon by running the following command:

systemctl status doca-xplane

Example output:

● doca-xplane.service - DOCA XPlane
     Loaded: loaded (/lib/systemd/system/doca-xplane.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2026-04-23 11:40:08 IDT; 12h ago
   Main PID: 71852 (docker)
...

5.2. Service Logs

DOCA XPlane service logs are permanently stored and can be reviewed at /var/log/xplane/.

5.3. Using the doca-xplane-client

DOCA XPlane ships with a dedicated CLI client (doca-xplane-client) that retrieves topology and observability data via gRPC. The following are examples of how to query the client (outputs have been trimmed for brevity).

5.3.1. Retrieve Overall System Status

Command:

$ doca-xplane-client get-status

JSON response:

{
  "serviceName": "xplane",
  "version": "0.1.0",
  "process": {
    "state": "RUNNING",
    "uptimeSeconds": "86400"
  },
  "topologySummary": {
    "groups": 2,
    "planes": 2
  },
  "health": {
    "planesUp": [1],
    "planesDegraded": [0],
    "overallState": "DEGRADED",
    "failures": {
      "localTotal": 1,
      "remoteTotal": 2
    }
  }
}

5.3.2. Print Topology as Read by DOCA XPlane Agent

Command:

$ doca-xplane-client get-topology

JSON response:

{
  "pfs": [
    {
      "plane": 0,
      "group": "rail0nic0",
      "pfName": "eth2",
      "pciAddress": "0000:08:00.0",
      "speedGbps": 200
    }
  ],
  "vfs": [
    {
      "vfRep": "eth2",
      "group": "rail0"
    }
  ],
  "patchPorts": []
}

5.3.3. Get Summary of Configured Planes

Command:

$ doca-xplane-client get-planes-summary

JSON response:

{
  "planes": [
    {
      "plane": 0,
      "state": "DEGRADED",
      "failures": {
        "localTotal": 1,
        "remoteTotal": 2
      },
      "traffic": {
        "packets": "1250000",
        "bytes": "900000000"
      }
    }
  ]
}

5.3.4. Get Local Failures (e.g., link down) for Specific Plane

Command:

$ doca-xplane-client get-plane-failures-local --plane_id 0

JSON response:

{
  "localFailures": [
    {
      "group": "rail0-nic0",
      "pfName": "eth2"
    }
  ]
}

5.3.5. Get Remote Failures for Specific Plane

Command:

$ doca-xplane-client get-plane-failures-remote --plane_id 0

JSON response:

{
  "remoteFailures": [
     {
      "cidr": "10.10.0.0/24",
      "groups": [
      ]
    },
    {
      "cidr": "2001:db8:10::/64",
      "groups": [
      ]
    },
     {
      "cidr": "0.0.0.0/0",
      "groups": [
        "2",
        "3"
      ]
    }, 
  ]
}

5.3.6. Get Traffic Summary for Specific Plane

Command:

$ doca-xplane-client get-plane-traffic-summary --plane_id 0

JSON response:

{
  "nonDiverted": {
    "packets": "1200000",
    "bytes": "864000000"
  },
  "divertedTo": {
    "packets": "50000",
    "bytes": "36000000"
  },
  "dropped": {
    "packets": "0",
    "bytes": "0"
  }
}

5.3.7. Get Statistics for Traffic Diverted from One Plane to Another

To get a complete picture of failover routing, you can query traffic diverted both from and to a specific plane:

Command for traffic diverted from the plane:

$ doca-xplane-client get-plane-traffic-diverted-from --plane_id 0

JSON response:

{
  "aggregated": {
    "packets": "75000",
    "bytes": "54000000"
  },
  "perPlane": {
    "1": {
      "packets": "75000",
      "bytes": "54000000",
      "percentage": 100
    }
  }
}

Command for traffic diverted to the plane:

$ doca-xplane-client get-plane-traffic-diverted-to --plane_id 0

JSON response:

{
  "aggregated": {
    "packets": "50000",
    "bytes": "36000000"
  },
  "perPlane": {
    "1": {
      "packets": "50000",
      "bytes": "36000000",
      "percentage": 100
    }
  }
}

Last updated: May 27, 2026