This guide describes how to use the NVIDIA DOCA™ XPlane service on hosts with NVIDIA® ConnectX®-8 or later SuperNICs.
Introduction
DOCA XPlane is a background service that manages local and remote plane failover in OVS-DOCA. It consumes route-availability updates from an NVIDIA® Spectrum-X fabric and programs OpenFlow rules for traffic failover. It also exposes gRPC telemetry for monitoring service health and topology state.
Requirements
To use the DOCA XPlane service, your environment must meet the following hardware and firmware prerequisites:
-
Hardware: A single-port ConnectX-8 or later SuperNIC.
-
Firmware version:
40.48.1000or later.
Host Configuration
Before you deploy the DOCA XPlane service, complete the following procedures.
Configure Firmware Settings on the SuperNIC
You must configure each SuperNIC: enable multi-plane mode, configure the number of planes, and set the number of physical functions (PFs) to match the number of planes.
-
Run the following
mlxconfigcommands to configure the SuperNIC:Bashsudo mlxconfig -d <mst-device> -y set SRIOV_EN=1 sudo mlxconfig -d <mst-device> -y set MODULE_SPLIT_M0[0..7]=1 MODULE_SPLIT_M0[8..15]=FF sudo mlxconfig -d <mst-device> -y set LAG_RESOURCE_ALLOCATION=1 sudo mlxconfig -d <mst-device> -y set LINK_TYPE_P1=2 sudo mlxconfig -d <mst-device> -y set LOAD_BALANCE_MODE_P1=2 sudo mlxconfig -d <mst-device> -y set NUM_OF_PF=4 NUM_OF_VFS=4 NUM_OF_PLANES_P1=4 -
To apply the configuration, gracefully shut down and power cycle the host.
Set devlink Parameters
The OVS-DOCA service must be stopped while you configure devlink parameters.
For each SuperNIC that DOCA XPlane manages, complete the following steps:
-
Get the PCI address of each PF. For example, run
lshw:root@server:~# lshw -class net -businfo Bus info Device Class Description ============================================================ ... pci@0000:9f:00.0 enp159s0f0np0 network CX8 Family [ConnectX-8] pci@0000:9f:00.1 enp159s0f1np1 network CX8 Family [ConnectX-8] pci@0000:9f:00.2 enp159s0f2np2 network CX8 Family [ConnectX-8] pci@0000:9f:00.3 enp159s0f3np3 network CX8 Family [ConnectX-8] ...In steps 2 through 4, replace the example PCI addresses with the addresses from step 1, in the form
pci/<address>. -
For each PF in that SuperNIC, set
flow_steering_modetohmfs:Bashsudo devlink dev param set pci/0000:9f:00.0 name flow_steering_mode cmode runtime value hmfs sudo devlink dev param set pci/0000:9f:00.1 name flow_steering_mode cmode runtime value hmfs sudo devlink dev param set pci/0000:9f:00.2 name flow_steering_mode cmode runtime value hmfs sudo devlink dev param set pci/0000:9f:00.3 name flow_steering_mode cmode runtime value hmfs -
For each PF in that SuperNIC, set the
eswitchmode toswitchdev:Bashsudo devlink dev eswitch set pci/0000:9f:00.0 mode switchdev sudo devlink dev eswitch set pci/0000:9f:00.1 mode switchdev sudo devlink dev eswitch set pci/0000:9f:00.2 mode switchdev sudo devlink dev eswitch set pci/0000:9f:00.3 mode switchdev -
For each PF in that SuperNIC, set
esw_multiporttotrue:Bashsudo devlink dev param set pci/0000:9f:00.0 name esw_multiport value true cmode runtime sudo devlink dev param set pci/0000:9f:00.1 name esw_multiport value true cmode runtime sudo devlink dev param set pci/0000:9f:00.2 name esw_multiport value true cmode runtime sudo devlink dev param set pci/0000:9f:00.3 name esw_multiport value true cmode runtime
Create SR-IOV Virtual Functions
For each SuperNIC that DOCA XPlane manages, create one virtual function (VF) on the first PF of the SuperNIC. In the following command, replace enp159s0f0np0 with the netdev name of that PF:
echo 1 | sudo tee /sys/class/net/enp159s0f0np0/device/sriov_numvfs > /dev/null
For more information about configuring SR-IOV, see HowTo Configure SR-IOV for ConnectX-4/ConnectX-5/ConnectX-6 with KVM (Ethernet).
Configure Hugepages and OVS-DOCA Settings
-
Use the doca-hugepages tool to provision at least 150 hugepages of 2048 KB for each SuperNIC that DOCA XPlane manages.
For example, on a server with 4 ConnectX-8 SuperNICs, run:
Bashsudo doca-hugepages config --app doca-xplane --size 2048 --num 600 sudo doca-hugepages reload -
Set the following Open vSwitch (OVS)
other_configoptions:Bashsudo ovs-vsctl set Open_vSwitch . other_config:doca-init=true sudo ovs-vsctl set Open_vSwitch . other_config:hw-offload=true sudo ovs-vsctl set Open_vSwitch . other_config:max-idle=300000 sudo ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true sudo ovs-vsctl set Open_vSwitch . other_config:pmd-sleep-max=50 -
Restart OVS-DOCA so that it uses the newly provisioned hugepages and settings:
Bash# Debian and derivatives sudo systemctl restart openvswitch-switch # RHEL and derivatives sudo systemctl restart openvswitch
XPlane Service
To deploy the DOCA XPlane service, follow these steps:
-
Install the
doca-xplanepackage from DOCA Host, available for download from the NVIDIA Developer website. -
After installation, populate the configuration file as described in the Configuration File section of this page.
-
After you save the configuration file, start the DOCA XPlane service by running the following command:
Bashsudo systemctl start doca-xplane
Command-line Client
DOCA XPlane ships with a command-line client, doca-xplane-client, that retrieves topology and observability data through gRPC.
To use the client, install the doca-xplane-client package from DOCA Host, available for download from the NVIDIA Developer website.
Configuration File
Before you start the service, you must populate the configuration file, /opt/mellanox/doca/services/xplane/xplane.toml, with the list of interfaces that the service uses.
The following table describes the settings that you can configure:
|
Setting Name |
Description |
Example Value |
|---|---|---|
|
|
Omit this setting, or set it to |
|
|
|
The MTU applied to all PFs that DOCA XPlane manages. The MTU that |
|
|
|
The group identifier for a set of PFs and an optional VF. Each group maps related interfaces that DOCA XPlane manages together. |
|
|
|
A map of PF netdev names to plane IDs. The PFs are attached to the |
|
|
|
The name of the VF representor interface that the group uses. |
|
|
|
The service logging level. Supported values are |
|
Example for GB200
[interfaces]
mtu = 9216
[interfaces.groups.rail0]
pfs_plane_map = {"eth_r0_p0" = 0, "eth_r0_p1" = 1, "eth_r0_p2" = 2, "eth_r0_p3" = 3}
vf_rep = "eth_vf_rep_r0"
[interfaces.groups.rail1]
pfs_plane_map = {"eth_r1_p0" = 0, "eth_r1_p1" = 1, "eth_r1_p2" = 2, "eth_r1_p3" = 3}
vf_rep = "eth_vf_rep_r1"
[interfaces.groups.rail2]
pfs_plane_map = {"eth_r2_p0" = 0, "eth_r2_p1" = 1, "eth_r2_p2" = 2, "eth_r2_p3" = 3}
vf_rep = "eth_vf_rep_r2"
[interfaces.groups.rail3]
pfs_plane_map = {"eth_r3_p0" = 0, "eth_r3_p1" = 1, "eth_r3_p2" = 2, "eth_r3_p3" = 3}
vf_rep = "eth_vf_rep_r3"
Example for VR200
This example describes a system with two 4-plane SuperNICs per GPU. Each PF connects to a different top-of-rack switch.
[interfaces]
mtu = 9216
[interfaces.groups.rail0_nic0]
pfs_plane_map = {"eth_r0_p0" = 0, "eth_r0_p1" = 1, "eth_r0_p2" = 2, "eth_r0_p3" = 3}
vf_rep = "eth_vf0_rep_r0"
[interfaces.groups.rail0_nic1]
pfs_plane_map = {"eth_r0_p4" = 4, "eth_r0_p5" = 5, "eth_r0_p6" = 6, "eth_r0_p7" = 7}
vf_rep = "eth_vf1_rep_r0"
[interfaces.groups.rail1_nic0]
pfs_plane_map = {"eth_r1_p0" = 0, "eth_r1_p1" = 1, "eth_r1_p2" = 2, "eth_r1_p3" = 3}
vf_rep = "eth_vf0_rep_r1"
[interfaces.groups.rail1_nic1]
pfs_plane_map = {"eth_r1_p4" = 4, "eth_r1_p5" = 5, "eth_r1_p6" = 6, "eth_r1_p7" = 7}
vf_rep = "eth_vf1_rep_r1"
[interfaces.groups.rail2_nic0]
pfs_plane_map = {"eth_r2_p0" = 0, "eth_r2_p1" = 1, "eth_r2_p2" = 2, "eth_r2_p3" = 3}
vf_rep = "eth_vf0_rep_r2"
[interfaces.groups.rail2_nic1]
pfs_plane_map = {"eth_r2_p4" = 4, "eth_r2_p5" = 5, "eth_r2_p6" = 6, "eth_r2_p7" = 7}
vf_rep = "eth_vf1_rep_r2"
[interfaces.groups.rail3_nic0]
pfs_plane_map = {"eth_r3_p0" = 0, "eth_r3_p1" = 1, "eth_r3_p2" = 2, "eth_r3_p3" = 3}
vf_rep = "eth_vf0_rep_r3"
[interfaces.groups.rail3_nic1]
pfs_plane_map = {"eth_r3_p4" = 4, "eth_r3_p5" = 5, "eth_r3_p6" = 6, "eth_r3_p7" = 7}
vf_rep = "eth_vf1_rep_r3"
Maintenance Mode
To turn maintenance mode on or off for a PF, use doca-xplane-client. For installation instructions, see the Command-line Client section of this page.
To turn on maintenance mode for a PF, run:
sudo doca-xplane-client maintenance set-link-maintenance --pf_name <pf> --enabled true
To turn off maintenance mode for a PF, run:
sudo doca-xplane-client maintenance set-link-maintenance --pf_name <pf> --enabled false
To get the maintenance status of a PF, run:
sudo doca-xplane-client maintenance get-link-maintenance --pf_name <pf>
The response is similar to the following:
{
"enabled": false
}
Monitoring
Use the following commands and logs to monitor the DOCA XPlane service.
Check System Status
To check the status of the DOCA XPlane service, run the following command:
sudo systemctl status doca-xplane
The output is similar to the following:
● doca-xplane.service - DOCA XPlane
Loaded: loaded (/lib/systemd/system/doca-xplane.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2026-04-23 11:40:08 IDT; 12h ago
Main PID: 71852 (docker)
...
Service Logs
The DOCA XPlane service stores its logs permanently in /var/log/xplane/.
Using doca-xplane-client
The following examples show how to query the DOCA XPlane service with doca-xplane-client. The outputs are trimmed.
Get Overall System Status
To get the overall status of the service, run:
sudo doca-xplane-client get-status
The response is similar to the following:
{
"serviceName": "xplane",
"version": "0.1.0",
"process": {
"state": "RUNNING",
"uptimeSeconds": "86400"
},
"topologySummary": {
"groups": 2,
"planes": 2
},
"health": {
"planesUp": [1],
"planesDegraded": [0],
"overallState": "DEGRADED",
"failures": {
"localTotal": 1,
"remoteTotal": 2
}
}
}
Get Topology as Read by DOCA XPlane
To get the topology as DOCA XPlane reads it, run:
sudo doca-xplane-client get-topology
The response is similar to the following:
{
"pfs": [
{
"plane": 0,
"group": "rail0nic0",
"pfName": "eth2",
"pciAddress": "0000:08:00.0",
"speedGbps": 200
}
],
"vfs": [
{
"vfRep": "eth2",
"group": "rail0"
}
],
"patchPorts": []
}
Get Summary of Configured Planes
To get a summary of the configured planes, run:
sudo doca-xplane-client get-planes-summary
The response is similar to the following:
{
"planes": [
{
"plane": 0,
"state": "DEGRADED",
"failures": {
"localTotal": 1,
"remoteTotal": 2
},
"traffic": {
"packets": "1250000",
"bytes": "900000000"
}
}
]
}
Get Local Failures for a Specific Plane
Local failures include link-down events. To get the local failures for a specific plane, run:
sudo doca-xplane-client get-plane-failures-local --plane_id 0
The response is similar to the following:
{
"localFailures": [
{
"group": "rail0-nic0",
"pfName": "eth2"
}
]
}
Get Remote Failures for a Specific Plane
To get the remote failures for a specific plane, run:
sudo doca-xplane-client get-plane-failures-remote --plane_id 0
The response is similar to the following:
{
"remoteFailures": [
{
"cidr": "10.10.0.0/24",
"groups": []
},
{
"cidr": "2001:db8:10::/64",
"groups": []
},
{
"cidr": "0.0.0.0/0",
"groups": [
"2",
"3"
]
}
]
}
Get Traffic Summary for a Specific Plane
To get the traffic summary for a specific plane, run:
sudo doca-xplane-client get-plane-traffic-summary --plane_id 0
The response is similar to the following:
{
"nonDiverted": {
"packets": "1200000",
"bytes": "864000000"
},
"divertedTo": {
"packets": "50000",
"bytes": "36000000"
},
"dropped": {
"packets": "0",
"bytes": "0"
}
}
Get Statistics for Traffic Diverted From One Plane to Another
To see failover routing in both directions, query the traffic diverted from and to a specific plane.
To get the traffic diverted from the plane, run:
sudo doca-xplane-client get-plane-traffic-diverted-from --plane_id 0
The response is similar to the following:
{
"aggregated": {
"packets": "75000",
"bytes": "54000000"
},
"perPlane": {
"1": {
"packets": "75000",
"bytes": "54000000",
"percentage": 100
}
}
}
To get the traffic diverted to the plane, run:
sudo doca-xplane-client get-plane-traffic-diverted-to --plane_id 0
The response is similar to the following:
{
"aggregated": {
"packets": "50000",
"bytes": "36000000"
},
"perPlane": {
"1": {
"packets": "50000",
"bytes": "36000000",
"percentage": 100
}
}
}
Last updated: