Remote Direct Memory Access (RDMA) enables direct memory-to-memory data transfers between servers without involving the CPU. RDMA over Converged Ethernet (RoCE) extends RDMA to lossless Ethernet networks and provides high-throughput, low-latency communication.
The NVIDIA® ConnectX® Ethernet adapter family supports RDMA transport over Ethernet. By offloading RDMA services to hardware, these adapters reduce CPU load and latency for performance-critical applications such as financial systems, databases, storage, and content delivery networks.
Key Considerations for RoCE
When you deploy RDMA applications over Ethernet, consider the following:
-
No Subnet Manager (SM): Unlike InfiniBand, RoCE does not require an SM. However, certain operations, such as multicast group joins, must be handled differently, even though the API remains unchanged.
-
No LID (Local ID): Because the LID is specific to InfiniBand, it is not set for RoCE ports and appears as zero in queries.
-
No APM (Alternate Path Migration): RoCE does not support APM because APM relies on InfiniBand-specific mechanisms.
-
Manual path record setup: Because you cannot query paths without an SM, you must populate the path record structure manually before you establish a connection. We strongly recommend using the RDMA communication manager (RDMA CM) because it automates path record handling.
-
Priority mapping through VLAN tags: VLAN-tagged Ethernet frames use a 3-bit priority field, the Priority Code Point (PCP), derived from the 3 least significant bits of the InfiniBand Service Level (SL) field.
-
RoCE traffic counters: The hardware offloads RoCE traffic, so the traffic is not visible in standard Ethernet interface counters. Instead, find RoCE traffic statistics in the following directory:
/sys/class/infiniband/DEVICE/ports/PORT/counters/Replace
DEVICEwith the RDMA device name, such asmlx5_0, andPORTwith the port number.
RoCE Modes
RoCE encapsulates InfiniBand transport in one of the following Ethernet packets:
-
RoCEv1: Uses a dedicated EtherType (
0x8915) -
RoCEv2: Encapsulates RDMA in UDP/IP packets, using UDP port
4791
The following diagram shows the RoCEv1 and RoCEv2 protocol stack:
RoCEv1
RoCEv1 uses a dedicated Ethernet EtherType (0x8915). It can include or omit VLAN tags and adheres to the standard Ethernet MTU. It is a Layer 2 protocol and is limited to communication within the same Ethernet broadcast domain.
RoCEv2
RoCEv2 extends RDMA to Layer 3 networks by encapsulating RDMA traffic in UDP/IP packets. This allows traffic to traverse IP routers and use standard IP routing. Key features include:
-
IP header: Enables routing across IP networks.
-
UDP header: Used for stateless encapsulation; destination port
4791. -
Source port field: Can serve as an opaque flow ID to assist with Equal-Cost Multi-Path (ECMP) routing.
Because the packet format change occurs at the wire level and does not affect the RDMA API, applications can operate over RoCEv1 or RoCEv2 without modification.
Both RoCEv1 and RoCEv2 are supported by default. The driver associates Global Identifier (GID) indexes with both modes, creating one entry per version.
For deployment examples, see the Recommended Network Configuration Examples For RoCE Deployment community post.
GID Table Population
GID table entries are automatically created when an IP address is configured on an Ethernet device associated with a RoCE port. Each GID entry includes:
-
GID value
-
GID type (for example, RoCEv1 or RoCEv2)
-
Associated network device
By default, each RoCE-capable port has two GID entries with the same value but different types (RoCEv1 and RoCEv2). A GID can have one of the following formats:
-
IPv4 GID: An IPv4-mapped IPv6 address
-
IPv6 GID: The IPv6 address itself
The Layer 3 header depends on the GID type and format:
-
RoCEv2 with an IPv4 GID: An IPv4 header
-
RoCEv2 with an IPv6 GID: An IPv6 header
-
RoCEv1, with an IPv4 or IPv6 GID: An InfiniBand Global Route Header (GRH)
View the GID Table in sysfs
The GID table is exposed to userspace through sysfs and provides details for each GID entry associated with a RoCE-capable port:
-
GID value:
/sys/class/infiniband/DEVICE/ports/PORT/gids/INDEX -
GID type (for example, RoCEv1 or RoCEv2):
/sys/class/infiniband/DEVICE/ports/PORT/gid_attrs/types/INDEX -
Associated network device:
/sys/class/infiniband/DEVICE/ports/PORT/gid_attrs/ndevs/INDEX
In each directory, each filename is a GID index, and the file contains the GID value, GID type, or network device for that index.
Set the RoCE Mode for a QP
On devices that support both RoCEv1 and RoCEv2, how you select the mode depends on the Queue Pair (QP) type:
RC and UC QPs (Connected QPs)
Reliable Connected (RC) and Unreliable Connected (UC) QPs are connected QPs. To transition a connected QP from the INIT state to the Ready to Receive (RTR) state, you must provide an Address Vector (AV). Among its attributes, the AV must include the GID index from the port’s GID table to define the source GID for the QP.
The GID type at the specified index determines the RoCE mode (v1 or v2) for the QP.
UD QPs (Unconnected QPs)
Unreliable Datagram (UD) QPs are connectionless and use GIDs for addressing. As with connected QPs, the GID index that you set in the AV determines the RoCE version used to send packets.
Set the RoCE Mode for RDMA CM Applications
The RDMA CM interface simplifies connection management by requiring only the active side to specify the IP address of the passive peer.
-
The RDMA CM stack selects a source GID from the local GID table.
-
Because multiple entries can have the same GID value but different types, the GID type must be specified for an accurate lookup.
Configure the RDMA CM GID Type
RDMA CM uses a default RoCE mode for each device port to determine which GID type to use when it resolves the local GID. To view or change this setting, use the cma_roce_mode utility:
-
To print the current RoCE mode for a port:
cma_roce_mode -d DEVICE -p PORT -
To set the RoCE mode (1 for RoCEv1, 2 for RoCEv2):
cma_roce_mode -d DEVICE -p PORT -m {1|2}
This setting affects RDMA CM applications by defining how the source GID is selected during connection establishment.
GID Table Example
The following table shows an example of a GID table:
|
Device |
Port |
Index |
GID |
IPv4 |
Type |
Netdev |
|---|---|---|---|---|---|---|
|
|
1 |
0 |
|
|
v1 |
|
|
|
1 |
1 |
|
|
v2 |
|
|
|
1 |
2 |
|
|
v1 |
|
|
|
1 |
3 |
|
|
v2 |
|
|
|
1 |
0 |
|
|
v1 |
|
|
|
1 |
1 |
|
|
v2 |
|
In this example:
-
On each device, indexes 0 and 1 on port 1 are the default GIDs, one for each supported RoCE type.
-
On
mlx5_0, indexes 2 and 3 on port 1 belong to IP address 10.10.10.1 onp4p1.
In general, packets from a QP that is associated with a GID of a VLAN interface have a VLAN header with the VLAN ID (VID) of that interface, and packets from a QP that is associated with a RoCEv2 IPv6 GID have an IPv6 header.
RoCE Lossless Ethernet Configuration
To function reliably, RoCE requires a form of flow control. You can use global flow control, but it is usually undesirable for performance reasons.
The recommended way to use RoCE is with Priority Flow Control (PFC). To use PFC, you must enable it on all endpoints and switches in the flow path.
Install and Load the Driver
To install and load the driver:
-
Install DOCA-Host. For more information, see DOCA Installation Guide for Linux. RoCE is installed as part of the mlx5 and other modules when you install the driver.
The list of the modules that will be loaded automatically upon boot can be found in the configuration file
/etc/infiniband/openib.conf. -
Display the installed driver version:
ofed_info -s -
Query the device information:
ibv_devinfo
In the ibv_devinfo output, check the following fields:
-
state:PORT_ACTIVEmeans that the port is active. To read the port state from sysfs, runcat /sys/class/infiniband/DEVICE/ports/PORT/state. -
link_layer:Ethernetmeans that the port uses Ethernet. To read the link layer from sysfs, runcat /sys/class/infiniband/DEVICE/ports/PORT/link_layer. -
fw_ver: The firmware version. To read it from sysfs, runcat /sys/class/infiniband/DEVICE/fw_ver.
Configure and Test RoCE
The following sections describe how to configure and test RoCE after you install and load the driver.
Associate InfiniBand Ports With Ethernet Ports
The mlx5_ib driver holds a reference to the net device to get notifications about the port state. It also uses the mlx5_core driver to resolve IP addresses to the MAC addresses that are required to create address vectors. However, RoCE traffic does not go through the mlx5_core driver; the hardware offloads it completely.
To see which Ethernet interface each InfiniBand port is associated with, run ibdev2netdev:
# ibdev2netdev
mlx5_0 port 1 <===> eth2
#
Configure an IP Address on the Netdev Interface
To configure an IP address on the netdev interface:
-
Configure an IP address on the netdev interface on both sides of the link, and bring the interface up:
# ip addr add 20.4.3.220/8 dev eth2 # ip link set dev eth2 up -
Make sure that
pingworks:ping 20.4.3.219 PING 20.4.3.219 (20.4.3.219) 56(84) bytes of data. 64 bytes from 20.4.3.219: icmp_seq=1 ttl=64 time=0.873 ms 64 bytes from 20.4.3.219: icmp_seq=2 ttl=64 time=0.198 ms 64 bytes from 20.4.3.219: icmp_seq=3 ttl=64 time=0.167 ms --- 20.4.3.219 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.167/0.412/0.873/0.326 ms
Add VLANs
To add VLANs:
-
Make sure that the
8021qmodule is loaded:modprobe 8021q -
Add a VLAN interface:
# ip link add link eth2 name eth2.7 type vlan id 7 -
Configure an IP address on the VLAN interface, and bring the interface up:
ip addr add 7.4.3.220/8 dev eth2.7 ip link set dev eth2.7 up
Define Ethernet Priority (PCP in 802.1Q Headers)
To define the Ethernet priority, run ibv_rc_pingpong on the server and on the client with the following options:
-
-l: The SL. The 3 least significant bits of the SL become the PCP. -
-g: The local GID index. The PCP is carried only in VLAN-tagged frames, so use a GID index that belongs to a VLAN interface. -
-i: The device port.
-
Define the Ethernet priority on the server:
# ibv_rc_pingpong -g 1 -i 2 -l 4 local address: LID 0x0000, QPN 0x1c004f, PSN 0x9daf6c, GID fe80::202:c900:708:e799 remote address: LID 0x0000, QPN 0x1c004f, PSN 0xb0a49b, GID fe80::202:c900:708:e811 8192000 bytes in 0.01 seconds = 4840.89 Mbit/sec 1000 iters in 0.01 seconds = 13.54 usec/iter -
Define the Ethernet priority on the client:
# ibv_rc_pingpong -g 1 -i 2 -l 4 sw419 local address: LID 0x0000, QPN 0x1c004f, PSN 0xb0a49b, GID fe80::202:c900:708:e811 remote address: LID 0x0000, QPN 0x1c004f, PSN 0x9daf6c, GID fe80::202:c900:708:e799 8192000 bytes in 0.01 seconds = 4855.96 Mbit/sec 1000 iters in 0.01 seconds = 13.50 usec/iter
Run RDMA CM Tests
-
Run the
ucmatosetest on the server:# ucmatose cmatose: starting server initiating data transfers completing sends receiving data transfers data transfers complete cmatose: disconnecting disconnected test complete return status 0 # -
Run the
ucmatosetest on the client:# ucmatose -s 20.4.3.219 cmatose: starting client cmatose: connecting receiving data transfers sending replies data transfers complete test complete return status 0 #
This server-client run has no PCP or VLAN because the IP address used does not belong to a VLAN interface. If you specify an IP address that belongs to a VLAN interface, the traffic goes over the VLAN.
Type of Service (ToS)
You can set the ToS field for RDMA CM sockets by using the rdma_set_option() API, the same way that you set it for regular sockets. If the ToS is not set, the default value (0) is used.
ToS Kernel Implementation
Modern Linux kernels (v2.6.39+) no longer use hard-coded bit shifting to determine service levels. Instead, the kernel uses a two-step mapping process:
-
ToS to Linux priority: The ToS byte is interpreted using RFC 1349 service bits to derive a Linux socket priority (
sk_prio). -
Linux priority to hardware resources: The Linux priority is then mapped to the InfiniBand SL and the VLAN PCP.
ToS to Linux Priority Mapping
The bit pattern extracted from the ToS value is mapped to a Linux priority as shown in the following table:
|
ToS Value |
Linux Priority |
Traffic Class |
|---|---|---|
|
0x0 to 0x6 |
0 |
Best effort |
|
0x8 to 0xE |
2 |
Bulk data |
|
0x10 to 0x16 |
6 |
Interactive |
|
0x18 to 0x1E |
4 |
Interactive bulk |
Linux Priority to SL and PCP Mapping
Once the Linux Priority is determined, the hardware driver maps it to specific network headers based on the interface configuration:
-
SL: The Linux priority is mapped to an SL based on the
tc mqprioconfiguration. Ifmqpriois not configured, the SL is derived from the VLAN device. If neither mapping is present, the SL defaults to 0. This mapping is independent of the hardware's priority-to-traffic-class (TC) mapping. -
VLAN PCP: For traffic on a tagged VLAN interface, the PCP is determined by the
tc mqprioconfiguration on the upper network device. Ifmqpriois not configured, the PCP is derived from the VLAN interface's egress map (configured viaip linkorvconfig).
DSCP
You can set a default ToS for RDMA CM QPs in the RDMA CM configfs. This lets you control the ToS field without changing your application's source code. To set the default ToS, create a directory for the device, and then write the ToS value (0 to 255) to the default_roce_tos file of the port:
mkdir -p /sys/kernel/config/rdma_cm/DEVICE
echo TOS_VALUE > /sys/kernel/config/rdma_cm/DEVICE/ports/PORT/default_roce_tos
Applications that set the ToS explicitly with the rdma_set_option() API override the configfs value and continue to work as expected. For more information about Differentiated Services Code Point (DSCP) marking, see the HowTo Set Egress ToS/DSCP on RDMA CM QPs community post.
RoCE LAG
RoCE link aggregation (RoCE LAG) mimics Ethernet bonding for RDMA devices and is available only for dual-port adapters. This feature is supported on kernel versions 4.9 and later.
RoCE LAG mode is entered when both Ethernet interfaces are configured as a bond in one of the following modes:
-
active-backup(mode 1) -
balance-xor(mode 2) -
802.3ad(LACP, mode 4)
Any change to the bonding configuration that violates one of the preceding rules (that is, the bonding mode is not 1, 2, or 4, or the two Ethernet interfaces that belong to the same adapter are not the only members of the bond interface) results in exiting RoCE LAG mode and returning to the normal configuration of one RDMA device per port.
When RoCE LAG is enabled, there is one device named mlx5_bond_0 instead of two RDMA devices (mlx5_0 and mlx5_1).
For information about how to configure RoCE LAG, see the HowTo Configure RoCE over LAG (ConnectX-4/ConnectX-5/ConnectX-6) community post.
Disable RoCE
By default, RoCE is enabled on all mlx5 devices. When RoCE is enabled, the device treats all traffic to UDP port 4791 as RoCE traffic.
If you need only Ethernet (no RDMA) and want to forward traffic to UDP port 4791, you can disable RoCE through sysfs.
When RoCE is disabled, only Ethernet traffic is supported. Therefore, there are no GID tables, and only Raw Ethernet QPs are supported.
To disable RoCE, write 0 to roce_enable. To enable it, write 1:
echo {0|1} > /sys/devices/PCI_BUS_ADDRESS/roce_enable
To query the current RoCE state, run:
cat /sys/devices/PCI_BUS_ADDRESS/roce_enable
Enable or Disable RoCE on VMs Through VFs
By default, when you configure virtual functions (VFs) on the hypervisor, RoCE is enabled on all VFs, so they require more OS memory from the virtual machine (VM). If you need only Ethernet (no RDMA) on the VM and want to save VM memory, you can disable RoCE on the VF from the hypervisor. Disabling RoCE also lets the VM use the RoCE UDP port (4791) for standard UDP traffic.
For information about how to enable or disable RoCE on a VF, see the HowTo Enable/Disable RoCE on VMs via VFs community post.
Force DSCP
This feature enables setting a global traffic_class value for all RC QPs, or setting a specific traffic class based on several matching criteria.
-
To set a single global traffic class to be applied to all QPs, write the desired global
traffic_classvalue to/sys/class/infiniband/DEVICE/tc/PORT/traffic_class.
Note the following:-
Negative values indicate that the feature is disabled.
traffic_classvalue can be set usingibv_modify_qp() -
Valid values are 0 to 255.
-
The ToS field is 8 bits, and the DSCP field is 6 bits. To set a DSCP value of X, multiply X by 4 (a 2-bit left shift). For example, to set a DSCP value of 24, set the ToS value to 96 (24 × 4 = 96).
-
To set multiple traffic class values based on the source IP address, the destination IP address, or both, write a rule to
/sys/class/infiniband/DEVICE/tc/PORT/traffic_class. For example:echo "tclass=16,src_ip=1.1.1.2,dst_ip=1.1.1.0/24" > /sys/class/infiniband/mlx5_0/tc/1/traffic_classAdding the
tclassprefix to the traffic class value is optional.
In the preceding example, traffic class 16 is set for any QP with source IP address 1.1.1.2 and a destination IP address in 1.1.1.0/24.
When you set a specific traffic class, the following rule precedence applies:
-
If a global traffic class value is set, it will be applied to all QPs
-
If no global traffic class value is set, and there is a rule with matching source and destination IPs applicable to at least one QP, it will be applied
-
Rules only with matching source and/or destination IPs have no defined precedence over other rules with matching source and/or destination IPs
Note the following about rules:
-
You can provide a mask for destination IPv4 addresses.
-
The order in which you insert rules does not affect rule precedence.
-
Overlapping rules are entirely up to the administrator.
-
Writing
tclass=-1removes the rule from the database.
Force Time to Live (TTL)
This feature sets a global Time to Live (TTL) value for all RC QPs.
To set it, write the TTL value to /sys/class/infiniband/DEVICE/tc/PORT/ttl. Valid values are 0 to 255.
Last updated: