Enhanced Connection Establishment

Introduction

Enhanced Connection Establishment (ECE) extends the IBTA-defined RDMA Connection Manager (RDMA-CM) layer. Introduced in IBTA version 1.4, ECE allows two Host Channel Adapters (HCAs) to negotiate optional transport features when establishing a queue pair (QP), on a per-flow basis. Because ECE operates at the connection-manager layer rather than the InfiniBand transport layer, it applies to both InfiniBand and RoCE fabrics.

ECE is supported on NVIDIA ConnectX-6 Dx and later devices.

ECE solves a fundamental problem in heterogeneous RDMA deployments. Features such as Selective Repeat and Congestion Control require both endpoints to agree before they are activated. Enabling a feature on only one endpoint, without the peer's knowledge, can cause packet drops, delays, or performance degradation. ECE provides a bilateral handshake through which the endpoints advertise and agree on supported features for each QP.

Negotiated Features

ECE supports negotiation of the following features:

ECE activates only the capabilities that both endpoints advertise. If one endpoint does not support a feature, ECE clears that feature from the agreed options and establishes the connection without it.

ECE Wire Format

Within RDMA-CM, ECE information is carried in the following connection-management messages:

• REQ

• REP

• SIDR_REQ

• SIDR_REP

The messages use the following fields:

• VendorID: Contains the IEEE organizationally unique identifier (OUI) for the vendor capability set.

• AttributeModifier: Contains the vendor-specific options supported by the endpoint.

At the ibverbs API level, these fields map to the ibv_ece structure defined in <infiniband/verbs.h>:

struct ibv_ece {
    uint32_t vendor_id;  /* IEEE OUI. NVIDIA = 0x15b3 */
    uint32_t options;    /* 4 MSB = ECE version; 28 LSB = feature bits */
    uint32_t comp_mask;
};

In the options field:

• The four most significant bits identify the ECE version.

• The remaining 28 bits represent negotiable capabilities as individual flags.

The version field allows ECE to support future extensions without breaking backward compatibility.

Connection Establishment Flow (RC QP)

For a reliable-connected QP, ECE negotiation occurs around the transition from INIT to RTR.

Both endpoints must call ibv_set_ece() before calling ibv_modify_qp() to transition the QP to RTR.

For applications that manage QP connection setup directly through ibverbs, the application must exchange the ECE values between endpoints through an out-of-band channel, such as a socket or shared memory.

When an application uses RDMA-CM, the CM stack carries the ECE values in its connection-management messages.

Features Negotiated via ECE

Selective Repeat

By default, InfiniBand transport uses go-back-N retransmission. When a packet is lost, the sender retransmits that packet and all subsequent packets, including packets that the receiver has already received.

Selective Repeat changes this behavior so that the sender retransmits only the missing packets. This reduces retransmission overhead in environments where packet loss occurs intermittently.

Selective Repeat can be enabled in either of the following ways:

• ECE negotiation – Enables Selective Repeat per QP after both endpoints advertise and agree on the capability. This approach is recommended for deployments containing different NIC generations.

• Device-wide configuration – Enables Selective Repeat for all QPs on the device by setting selective_repeat_forced_en in the ROCE_ACCL register. This method bypasses ECE negotiation.

Congestion Control

The CC feature bits in the ECE options negotiate which congestion-control algorithms are enabled on the QP, not merely whether congestion control is enabled. The negotiated result can include ZTR-RTT CC, DCQCN, or both.

Both endpoints must agree on each algorithm. An algorithm that is not supported by both endpoints is not activated for the QP.

NVIDIA NICs support the following built-in congestion-control algorithms:

• ZTR-RTT CC – Uses hardware-timestamped round-trip-time measurements and does not require specialized switch configuration. Both endpoints must negotiate ZTR-RTT CC before either endpoint begins sending RTT probe packets.

• DCQCN – Uses Explicit Congestion Notification (ECN) marking from ECN-capable switches.

For configuration information, see:

• ZTR-RTT CC Algorithm

• ZTR-RTT CC Parameter and Counter Configuration

• DCQCN CC Algorithm

• DCQCN CC Parameters

Integrating ECE

API Summary

The supported output parameter indicates whether the underlying driver supports ECE.

When supported is 0, the QP does not support ECE. The application can continue establishing the connection without activating ECE-negotiated features, and the feature set defaults to the device configuration.

Raw ibverbs Integration

Applications that manage QP connection setup directly must exchange ECE values through the same side channel used to exchange information such as the local identifier, QP number, and packet sequence number.

The following example shows the ECE negotiation sequence using an out-of-band socket.

Requestor

#include <infiniband/verbs.h>

/* Helper: send/receive ECE over an established socket fd */
static int send_ece(int fd, struct ibv_ece *ece) {
    uint32_t buf[2] = { htonl(ece->vendor_id), htonl(ece->options) };
    return write(fd, buf, sizeof(buf)) == sizeof(buf) ? 0 : -1;
}

static int recv_ece(int fd, struct ibv_ece *ece) {
    uint32_t buf[2];
    if (read(fd, buf, sizeof(buf)) != sizeof(buf))
        return -1;

    ece->vendor_id = ntohl(buf[0]);
    ece->options = ntohl(buf[1]);
    return 0;
}

int requestor_connect(struct ibv_qp *qp, int sockfd)
{
    struct ibv_ece local_ece = {};
    struct ibv_ece remote_ece = {};
    int supported = 0;
    int ret;

    /*
     * Step 1: After the QP is in INIT, retrieve the local ECE
     * capabilities.
     */
    ret = ibv_query_ece(qp, &local_ece, &supported);
    if (ret || !supported) {
        /* ECE is not supported. Continue without ECE negotiation. */
        goto move_to_rtr;
    }

    /*
     * Step 2: Send the local ECE capabilities to the responder through
     * the out-of-band socket.
     */
    if (send_ece(sockfd, &local_ece) < 0)
        return -1;

    /*
     * Step 3: Receive the reduced set of mutually supported ECE options
     * from the responder.
     */
    if (recv_ece(sockfd, &remote_ece) < 0)
        return -1;

    /*
     * Step 4: Apply the agreed ECE options before transitioning the QP
     * to RTR.
     */
    ret = ibv_set_ece(qp, &remote_ece, &supported);
    if (ret) {
        fprintf(stderr, "ibv_set_ece failed: %s
", strerror(ret));
        return -1;
    }

move_to_rtr:
    /* Step 5: Transition the QP to RTR. */
    struct ibv_qp_attr attr = {
        .qp_state = IBV_QPS_RTR,
        .path_mtu = IBV_MTU_4096,
        .dest_qp_num = remote_qpn, /* Received from the peer */
        .rq_psn = remote_psn,
        .max_dest_rd_atomic = 1,
        .min_rnr_timer = 12,
        /* Populate ah_attr from the remote LID or GID. */
    };

    return ibv_modify_qp(
        qp,
        &attr,
        IBV_QP_STATE |
        IBV_QP_AV |
        IBV_QP_PATH_MTU |
        IBV_QP_DEST_QPN |
        IBV_QP_RQ_PSN |
        IBV_QP_MAX_DEST_RD_ATOMIC |
        IBV_QP_MIN_RNR_TIMER);
}

Responder

int responder_accept(struct ibv_qp *qp, int sockfd)
{
    struct ibv_ece remote_ece = {};
    struct ibv_ece reduced_ece = {};
    int supported = 0;
    int ret;

    /* Step 1: Receive the requestor's ECE capabilities. */
    if (recv_ece(sockfd, &remote_ece) < 0)
        return -1;

    /*
     * Step 2: Apply the received ECE options. The driver calculates
     * the intersection with the responder's locally supported options.
     */
    ret = ibv_set_ece(qp, &remote_ece, &supported);
    if (ret || !supported) {
        /*
         * ECE is not supported on this endpoint. Return an empty ECE
         * result and continue without ECE-negotiated features.
         */
        memset(&reduced_ece, 0, sizeof(reduced_ece));
        goto send_reduced;
    }

    /*
     * Step 3: Retrieve the reduced set of mutually supported ECE
     * options.
     */
    ret = ibv_query_ece(qp, &reduced_ece, &supported);
    if (ret) {
        fprintf(stderr, "ibv_query_ece failed: %s
", strerror(ret));
        return -1;
    }

send_reduced:
    /* Step 4: Return the reduced ECE options to the requestor. */
    if (send_ece(sockfd, &reduced_ece) < 0)
        return -1;

    /*
     * Step 5: Transition the QP to RTR with the agreed features
     * active.
     */
    struct ibv_qp_attr attr = {
        .qp_state = IBV_QPS_RTR,
        /* Populate the remaining fields from the requestor's QP data. */
    };

    return ibv_modify_qp(
        qp,
        &attr,
        IBV_QP_STATE |
        IBV_QP_AV |
        IBV_QP_PATH_MTU |
        IBV_QP_DEST_QPN |
        IBV_QP_RQ_PSN |
        IBV_QP_MAX_DEST_RD_ATOMIC |
        IBV_QP_MIN_RNR_TIMER);
}

RDMA-CM Integration

When an application uses RDMA-CM, the CM stack manages the REQ and REP messages.

Set ECE on the CM-managed QP before calling rdma_connect() or rdma_accept(). RDMA-CM then carries the ECE options as part of its internal InfiniBand connection-management handshake.

Client

#include <rdma/rdma_cma.h>
#include <infiniband/verbs.h>

int client_connect(
    struct rdma_cm_id *id,
    struct rdma_conn_param *conn_param)
{
    struct ibv_ece ece = {};
    int supported = 0;

    /*
     * The QP must already exist. Call rdma_create_qp() before this
     * function.
     */

    /*
     * Step 1: Retrieve the local ECE capabilities from the CM-managed
     * QP.
     */
    if (ibv_query_ece(id->qp, &ece, &supported) == 0 && supported) {
        /*
         * Optionally restrict the options to the features that the
         * application wants to activate.
         *
         * Leaving the options unchanged advertises all locally
         * supported features.
         */
    }

    /*
     * Step 2: rdma_connect() sends the IB REQ. The CM includes the QP's
     * current ECE options in the request.
     */
    return rdma_connect(id, conn_param);
}

Server

Handle ECE negotiation after receiving the CONNECT_REQUEST event and before calling rdma_accept().

int server_on_connect_request(struct rdma_cm_event *event)
{
    struct rdma_cm_id *child_id = event->id;
    struct ibv_ece remote_ece = {};
    struct ibv_ece reduced_ece = {};
    int supported = 0;

    /* Create a QP on the child CM ID before negotiating ECE. */
    struct ibv_qp_init_attr qp_attr = {
        /* Populate the QP initialization attributes. */
    };

    if (rdma_create_qp(child_id, pd, &qp_attr))
        return -1;

    /*
     * Step 1: The CM exposes the requestor's ECE through ibv_query_ece()
     * on the child QP after the QP is created and before rdma_accept().
     * Apply it with ibv_set_ece() to calculate the intersection.
     */
    if (ibv_query_ece(
            child_id->qp,
            &remote_ece,
            &supported) == 0 && supported) {
        ibv_set_ece(child_id->qp, &remote_ece, &supported);

        /*
         * Retrieve the reduced set of mutually supported ECE options.
         */
        ibv_query_ece(
            child_id->qp,
            &reduced_ece,
            &supported);
    }

    /*
     * Step 2: rdma_accept() sends the IB REP containing the reduced
     * ECE options.
     */
    struct rdma_conn_param accept_param = {
        /* Populate the connection parameters. */
    };

    return rdma_accept(child_id, &accept_param);
}

When the client receives the REP, RDMA-CM applies the reduced ECE options to the client QP automatically. The client does not need to call ibv_set_ece() explicitly after receiving the response.

Backward Compatibility

ECE is designed to allow connections to proceed when the endpoints support different capabilities.

The following fallback behavior applies:

• Peer without ECE support – If ibv_query_ece() sets supported to 0, or the remote endpoint sends a vendor_id value of 0, the connection proceeds without ECE-negotiated features, and the feature set defaults to the device configuration.

• Partial feature mismatch: ECE clears any feature that one endpoint does not support. The connection proceeds with only the capabilities supported by both endpoints.

• ECE mismatch – NVIDIA NICs do not send a REJ response because of an ECE mismatch. The connection falls back to the mutually supported feature set. This is graceful degradation, not connection failure.

Platform Support

ECE is supported on NVIDIA ConnectX-6 Dx and later devices.

Firmware and InfiniBand device-verbs support for ECE predates adoption by many applications. An application that does not call ibv_query_ece() and ibv_set_ece() does not activate ECE-gated features on its QPs.

Version History

RDMA-CM and Linux Kernel

rdma-core

NCCL

Related Documentation

• ZTR-RTT CC Algorithm

• ZTR-RTT CC Parameter and Counter Configuration

• DCQCN CC Algorithm

• DCQCN CC Parameters

Last updated: September 01, 2026