Networking DOCA SDK Documentation

DOCA Flow Connection Tracking

This guide provides an overview and configuration instructions for the DOCA Flow Connection Tracking (CT) API.

Introduction

The DOCA Flow Connection Tracking (CT) module is a 5-tuple table designed to efficiently track network connections using hardware resources. It supports the following key features:

  • Track zone and 5-tuple sessions – Track and manage network connections based on a 5-tuple (source IP, destination IP, source port, destination port, and protocol) along with zone-based separation

  • IPv4 and IPv6 sessions – Track connections for both IP versions; each direction can independently use IPv4 or IPv6

  • Zone-based virtual tables – Enable logical isolation using zones

  • Aging support – Remove idle connections automatically using configurable timeouts

  • Connection metadata – Set and manage metadata for tracked connections

  • Bidirectional packet handling – Manage traffic in both directions of a connection

  • TCP window validation – Validate per-connection TCP state in hardware and expose a packet syndrome for downstream processing

  • High connection rate – Efficiently handle a high rate of connections per second (CPS)

The CT module makes it simple and efficient to track connections by leveraging hardware resources. 

Architecture

The DOCA Flow CT Pipe is designed to handle non-encapsulated TCP and UDP packets. It supports two primary actions:

  • Forward to next pipe – For packets that match a known 6-tuple connection (5-tuple + zone)

  • Miss to next pipe – For packets without a matching connection entry

The application is responsible for handling packets based on these outcomes.

The DOCA Flow CT API consists of four major components:

  • CT module manipulation – Configure and manage resources within the CT module

  • CT connection entry manipulation – Add, remove, or update connection entries efficiently

  • Callbacks – Handle asynchronous processing results for connection entries

  • Pipe and entry statistics – Monitor connection tracking performance using pipe-level and entry-level statistics

These components provide flexible control over connection tracking and monitoring, allowing applications to adapt to various network scenarios effectively.

arch-diagram.png

Aging

Aging time refers to the maximum duration (in seconds) a session can remain active without detecting any packets. If no packets are observed within this period, the session is terminated.

To support aging, aging checks run on the counter service thread, which polls and checks counters for all active connections each cycle, ensuring that stale sessions are removed efficiently.

When aging is enabled, either the counter flag or a non-zero timeout must be set for at least one connection entry to trigger session expiration.

Application Responsibilities

The application is responsible for:

  • Managing worker threads

  • Parsing and handling connection lifecycles

Applications use DOCA Flow CT management APIs for creating and destroying connections.

The CT aging module automatically notifies the application of aged-out connections by invoking callbacks.

Connection Rules and Management

Users have the flexibility to create connection rules with different patterns, metadata (meta), or counters which can be applied separately for each packet direction.

Users must manually define the appropriate meta and mask values for matching (match) and modifying (modify) packets.

To create rules in stages:

  1. Create one rule for a connection using the standard API.

  2. Add a second rule for the opposite packet direction using the doca_flow_ct_entry_add_dir() API.

connection-rules-diagram.png

Processing CT Entries

DOCA Flow provides specialized APIs to process CT entries using a dedicated queue:

  • doca_flow_ct_entries_process – Processes CT pipe entries in the queue

  • doca_flow_aging_handle – Handles the aging of pipe entries

  • doca_flow_ct_query_entry – Queries per-connection packet/byte statistics.

Do not call doca_flow_ct_entries_process() or doca_flow_entries_process() on the same CT queue from an entry completion callback, the TCP window-validation completion callback, or a doca_flow_ct_pipe_iterate() callback. A nested call returns DOCA_ERROR_AGAIN without consuming the outer call's completions. Defer processing until the outer call returns rather than retrying inside the callback.

Some APIs, such as CT entry status queries and pipe miss queries, are not supported.

CT Pipe Configuration

CT pipes extend standard pipe creation with additional optional doca_flow_pipe_cfg parameters:

Function

Description

doca_flow_pipe_cfg_set_ct_connections()

Sets the maximum number of IPv4, IPv6, and total CT connections.

doca_flow_pipe_cfg_set_ct_max_connections_per_zone()

Sets the maximum CT connections permitted per zone.

doca_flow_pipe_cfg_set_ct_dup_filter_size()

Sets the number of connections to cache in the duplication filter.

IPv6 Support

DOCA Flow CT tracks IPv6 connections as well as IPv4, including asymmetric flows where origin and reply use different IP versions (e.g. IPv4 origin, IPv6 reply). IPv4 and IPv6 sessions use separate pools (sized via doca_flow_pipe_cfg_set_ct_connections()); per entry, the IPv6 origin/reply flags in doca_flow_ct.h select which direction matches on IPv6.

CT IPv6 steering is hardware-dependent - supported on BlueField-3 / ConnectX-8 and above (BlueField-2 / ConnectX-7 are IPv4-only). Query support before allocating IPv6 sessions; if unsupported, set the IPv6 session count to 0 in doca_flow_pipe_cfg_set_ct_connections().

Querying CT IPv6 Support

Use the port capabilities API (doca_flow_port_capabilities_create / _get_ct_ipv6 / _destroy) to check device support before configuring CT. Perform the create -> query -> destroy cycle after doca_flow_init() and before port start (see Port Capabilities in the DOCA Flow programming guide). These APIs are DOCA_EXPERIMENTAL - build with -DDOCA_ALLOW_EXPERIMENTAL_API.

See the doca_flow_ct_tcp_ipv4_ipv6 sample for an end-to-end example, including the capability query.

Prerequisites

Connection tracking is unsupported with application-defined DOCA Flow definitions (DOCA_FLOW_DEFINITIONS_MODE_APP_DEFINED): doca_flow_init() rejects CT queues in that mode with DOCA_ERROR_NOT_SUPPORTED. Use the default library-defined definitions when CT is required.

DPU

To enable DOCA Flow CT on the DPU, perform the following on the Arm:

  1. Enable iommu.passthrough in Linux boot commands (or disable SMMU from the DPU BIOS): 

    1. Run:

      sudo vim /etc/default/grub
      

      Set GRUB_CMDLINE_LINUX="iommu.passthrough=1".

    2. Run: 

      sudo update-grub
      sudo reboot
      
  2. Configure DPU firmware with LAG_RESOURCE_ALLOCATION=1:

    sudo mlxconfig -d <device-id> s LAG_RESOURCE_ALLOCATION=1
    

    Retrieve device-id from the output of the mst status -v command. If, under the MST tab, the value is N/A, run the mst start command.

  3. Update /etc/mellanox/mlnx-bf.conf as follows:

    ALLOW_SHARED_RQ="no"
    
  4. Perform power cycle on the host and Arm sides.

  5. If working with a single port, set the DPU into e-switch mode: 

    sudo devlink dev eswitch set pci/<pcie-address> mode switchdev
    sudo devlink dev param set pci/<pcie-address> name esw_multiport value false cmode runtime
    

    Retrieve pcie-address from the output of the mst status -v command.

  6. If working with two PF ports, set the DPU into multi-port e-switch mode (for the 2 PCIe devices):

    sudo devlink dev param set pci/<pcie-address> name esw_multiport value true cmode runtime
    

    Retrieve pcie-address from the output of the mst status -v command.

ConnectX

To enable DOCA Flow CT on the NVIDIA® ConnectX®, perform the following:

  1. Configure firmware with LAG_RESOURCE_ALLOCATION=1:

    sudo mlxconfig -d <device-id> s LAG_RESOURCE_ALLOCATION=1
    

    Retrieve device-id from the output of the mst status -v command. If, under the MST tab, the value is N/A, run the mst start command.

  2. Perform power cycle.

  3. If working with a single port:

    sudo devlink dev eswitch set pci/<pcie-address> mode switchdev
    sudo devlink dev param set pci/<pcie-address> name esw_multiport value false cmode runtime
    

    Retrieve pcie-address from the output of the mst status -v command.

  4. If working with two PF ports:

    sudo devlink dev eswitch set pci/<pcie-address0> mode switchdev
    sudo devlink dev eswitch set pci/<pcie-address1> mode switchdev
    sudo devlink dev param set pci/<pcie-address0> name esw_multiport value true cmode runtime
    sudo devlink dev param set pci/<pcie-address1> name esw_multiport value true cmode runtime
    

    Retrieve pcie-address from the output of the mst status -v command.

Actions

DOCA Flow CT supports actions based on meta and NAT operations. Each action can be defined as either shared or non-shared.

Action descriptors are not supported in CT action templates or CT entry actions.

Shared Actions

Actions that can be shared between entries. Shared actions are predefined and reused in multiple entries.

The user gets a handle per shared action created and uses this handle as a reference to the action where required. 

It is user responsibility to track shared actions and to remove them when they become irrelevant.

Shared actions are defined using a control queue (see doca_flow_cfg_set_ct_ctrl_queues()).

Non-shared Actions

Actions provided with their data during entry create/update.

These actions are completely managed by DOCA Flow CT and cannot be reused in multiple flows (i.e., NAT operations).

Action Sets in Pipe Creation

When creating a DOCA Flow CT pipe, users must define action sets, just as they would for any other pipe.

Fields in the CT pipe must be marked as CHANGEABLE during pipe creation. This allows the actual criteria for these fields to be specified later during entry creation.

Only actions related to meta and NAT, as defined in struct doca_flow_ct_actions, are supported.

During entry creation or update, different actions can be specified for each direction, allowing variations in action content and/or action type.

Feature Enable

To enable user actions, administrators must configure the following parameters:

  • User action templates must be configured during the DOCA Flow CT pipe creation phase.

    The maximum memory allocated for user actions (actions_mem_size) must be set on the CT-hosting port's doca_flow_port_cfg before it starts.

Using Actions

Init

Before calling doca_flow_init(), you must configure the following parameters on doca_flow_cfg:

  • ct_ctrl_queues (doca_flow_cfg_set_ct_ctrl_queues()): The total number of control queues dedicated to defining shared actions.

Before starting the CT-hosting port, you must configure the following on its doca_flow_port_cfg:

  • ct_actions_mem_size (doca_flow_port_cfg_set_ct_actions_mem_size()): The maximum amount of memory (in bytes) allocated for user actions. This value must be strictly 64-byte aligned, and NVIDIA highly recommends utilizing a power of 2.

Create DOCA Flow CT Pipe

Configure actions sets on doca_flow_pipe_create().

Create Shared Actions

Use doca_flow_ct_actions_add_shared() with one of the control queues.

Shared actions can be added at any time before use.

Add Entry

Entry can be created in one of the following ways:

  • Using an action handle of a predefined shared action

  • Using action data, which is specific to the flow and not sharable (for example, NAT operations)

The entry can have different actions and/or different action types per direction.

Remove Entry

Non-shared actions associated with an entry are implicitly destroyed by DOCA Flow CT.

Shared actions are not destroyed. They can be used by the user until they decide to remove them.

Update Entry

Entry actions can be updated per direction. All combinations of shared/non-shared actions are applicable (e.g., update from shared to non-shared).

Changeable Forward

DOCA Flow CT permits the use of a different forward pipe for each flow direction. The module operates at one of two mutually exclusive forwarding levels:

  • Pipe level – A single forward pipe is defined during DOCA Flow CT pipe creation and applies to all entries universally.

  • Entry level – The forward pipe is defined dynamically during entry creation.

Entry-level forwarding characteristics:

  • It exclusively supports DOCA_FLOW_FWD_PIPE and DOCA_FLOW_FWD_ORDERED_LIST_PIPE (fixed pipe, changeable index).

  • It supports defining a distinct forward pipe per flow direction (both directions can utilize the same or different forward pipes).

  • Because there is no default forward pipe, forwarding destinations must be explicitly set upon each entry creation.

Important

Changeable forwarding is not supported on a port configured with a non-zero DOCA_FLOW_RESOURCE_TCP_WINDOW_VALIDATION resource count. Every CT pipe on such a port must use a fixed pipe-level destination.

Enabling Changeable Forwarding

To enable this feature, create the DOCA Flow CT pipe using one of the following configurations:

  • Standard pipe:

    • Set forward type to DOCA_FLOW_FWD_PIPE 

    • Set next_pipe to NULL 

  • Ordered list pipe:

    • Set forward type to DOCA_FLOW_FWD_ORDERED_LIST_PIPE 

    • Set ordered_list_pipe.pipe to <ol_pipe> 

    • Set ordered_list_pipe.idx to UINT32_MAX 

Using Changeable Forward

To utilize changeable forwarding, execute the following sequence:

  1. Initialize DOCA Flow with CT queues configured on doca_flow_cfg (doca_flow_cfg_set_ct_queues() / doca_flow_cfg_set_ct_ctrl_queues()), then start the CT-hosting port.

  2. Create pipe by calling doca_flow_pipe_create() using the changeable forwarding configurations described above.

  3. Add entry by calling doca_flow_ct_add_entry(). During this step, set fwd_origin and/or fwd_reply to your desired targets.

  4. Update entry by calling doca_flow_ct_update_entry() to update the forwarding for a specific entry direction.

When updating the forward destination, you must explicitly pass all other parameters with their previously existing values.

Entry Iterator

When iterator support is enabled, DOCA Flow CT can traverse all entries on a CT pipe using a registered callback. For each invocation, the application retrieves full entry data (match, hash, flags) via doca_flow_ct_get_entry() and can recreate or mirror those entries.

A primary use case for this is High-Availability (HA) Synchronization: the application reads every active entry from the CT on the active node and programs matching entries on the standby node to preserve connection state during failovers. Iteration is incremental; the application drives progress by calling doca_flow_ct_entries_process() per queue, and the registered callback executes as entries are dispatched.

Enabling Iterator

  1. Create or configure the CT with the DOCA_FLOW_CT_FLAG_ITERATOR flag. 

    The CT duplication filter is backed by a hash table of active entries to prevent duplicate insertions while iteration and forwarding run concurrently.

  2. Start pipe-level iteration by calling doca_flow_ct_pipe_iterate(ct_pipe, iterate_cb, iterate_usr_ctx). This schedules the iteration across all queues for the given pipe.

  3. For each CT queue, call doca_flow_ct_entries_process() and pass the max_processed_entries limit. This processes the hardware steering queue and invokes iterate_cb as entries are delivered.

    • Inside the callback, read the entry details via doca_flow_ct_get_entry() to obtain the matcher, hash, and flags for standby replication.

    • If the number of processed entries returned is less than the requested max_processed_entries, the iteration for that specific queue has reached its end for the current pass.

  4. Pipe iteration formally completes once all participating queues have finished the incremental processing steps and no further callbacks are pending.

Iterator Limitations

  • Action exclusion: Entry actions are not exported through the iterator path. The application must manually retain or reconstruct CT entry actions on the standby node.

  • Incomplete passes: New entries created during or immediately after a walk are not guaranteed to be captured in a single iterator pass. Applications should track new entries independently and not rely solely on the iterator for complete HA synchronization.

Connection Tracking Offload

A CT entry can carry a connection tracking offload resource, which validates the TCP state of every packet that hits the entry and reports the outcome as a packet syndrome. The resource holds a profile — TCP state, per-direction window state, and the last packet seen — described by struct doca_flow_ct_tcp_window_validation.

The APIs in this section are experimental. The asynchronous completion callback and the syndrome matching interface require DOCA Flow ABI v3.

Configuring the Resource Pool

The TCP window-validation pool is an entry-bound per-port resource, not an ID-based shared resource. Configure it in this order:

  1. Before doca_flow_init(), call doca_flow_cfg_set_resource_mode() with DOCA_FLOW_RESOURCE_MODE_PORT and configure at least one CT queue with doca_flow_cfg_set_ct_queues().

  2. Call doca_flow_init().

  3. Before doca_flow_port_start(), call doca_flow_port_cfg_set_nr_resources() with DOCA_FLOW_RESOURCE_TCP_WINDOW_VALIDATION and the maximum number of entries that may use TCP window validation concurrently.

The pool is fixed when the port starts and cannot grow while the port is running. CT entry lifecycle APIs allocate and release these resources; the shared-resource get, put, configure, and query APIs do not apply. Requesting TCP window validation through doca_flow_ct_entry_prepare() on a pipe with no configured pool returns DOCA_ERROR_NOT_SUPPORTED. CT pipe creation on a representor can succeed, but preparing an entry that requests TCP window validation returns DOCA_ERROR_NOT_SUPPORTED. If no free validation resource remains for a new entry, prepare returns DOCA_ERROR_FULL.

Initializing a Validation Profile

An update writes a complete struct doca_flow_ct_tcp_window_validation profile, not a partial patch:

  • Multi-byte fields use host byte order.

  • Set enable to 1 to allow hardware updates. Setting it to 0 freezes the profile. Setting liberal_mode to 1 disables strict window validation in both directions.

  • state starts at DOCA_FLOW_CT_TCP_WINDOW_VALIDATION_STATE_TCP_SYN_RECV, after SYN-ACK; there is no state for a connection that has seen only SYN. last_index accepts one DOCA_FLOW_CT_TCP_WINDOW_VALIDATION_TCP_LAST_* enumerator, not an ORed combination.

  • max_ack_window accepts values from 0 through 7, where 0 selects the default 64 KiB ACK window. Each direction's scale accepts 0 through 0xF, where 0xF disables window scaling.

  • The last_*, live_connection, and per-direction state fields are maintained by hardware after traffic, but they are also writable. Seed them when adopting an existing connection. Leaving them zero in a later update resets that validation baseline, and an update overwrites values returned by an earlier query.

When converting a DPDK rte_flow_action_conntrack profile, copy fields by name instead of using memcpy or struct assignment. Invert the meaning of DPDK last_direction: DOCA Flow uses 0 for origin and 1 for reply.

Binding an Offload Resource to an Entry

  1. Call doca_flow_ct_entry_prepare() with both DOCA_FLOW_CT_ENTRY_FLAGS_ALLOC_ON_MISS and DOCA_FLOW_CT_ENTRY_FLAGS_TCP_WINDOW_VALIDATION. A resource is allocated only for a newly created entry and remains bound for that entry's lifetime. If conn_found is true, no new resource is allocated or retrofitted onto the existing entry.

  2. Call doca_flow_ct_tcp_window_validation_update() on the CT pipe with the entry and the initial profile. Complete this initial update before exposing the entry to traffic.

  3. Call doca_flow_ct_add_entry(). The per-entry fwd_origin and fwd_reply are overridden so matching traffic reaches the pipe-level next destination.

  4. Call doca_flow_ct_entries_process() as usual. The entry is reported only after both its rules and its profile update are in hardware, and a failed profile update fails the entry add.

The following pseudocode shows the required ordering in the default synchronous mode, with no TCP window-validation completion callback registered:

/* Synchronous mode: update returns only after the profile reaches hardware. */
prepare_flags = DOCA_FLOW_CT_ENTRY_FLAGS_ALLOC_ON_MISS |
                DOCA_FLOW_CT_ENTRY_FLAGS_TCP_WINDOW_VALIDATION;
result = doca_flow_ct_entry_prepare(queue, pipe, prepare_flags, /* ... */, &entry, &conn_found);
if (result == DOCA_SUCCESS && !conn_found) {
        result = doca_flow_ct_tcp_window_validation_update(pipe, queue, entry, &profile, true);
        if (result == DOCA_SUCCESS)
                result = doca_flow_ct_add_entry(queue, pipe, /* ... */, entry);
        if (result != DOCA_SUCCESS) {
                rollback_result = doca_flow_ct_entry_prepare_rollback(queue, pipe, entry);
                /* Handle rollback_result separately. */
        }
}

If the initial update or add call fails synchronously, call doca_flow_ct_entry_prepare_rollback() and handle any rollback error before discarding the entry handle.

If an add step fails after at least one rule direction was submitted, doca_flow_ct_add_entry() returns DOCA_SUCCESS; doca_flow_ct_entries_process() later reports one failed ADD callback while the library rolls back the submitted rules. Do not call prepare rollback for an add that returned success.

Skipping step 2 fails doca_flow_ct_add_entry() with DOCA_ERROR_BAD_STATE. A resource returned to the pool keeps the state of its previous connection, so an entry must never start tracking against a profile it did not write.

Between doca_flow_ct_add_entry() and the completion of an asynchronous profile update, hardware validates packets against the previous content of the resource. Letting the update complete before adding the entry closes that window; no_wait only shortens it, because the update completes on a queue independent of the one carrying the entry's rules.

Querying and Updating a Bound Profile

After an entry add and its initial profile write complete, doca_flow_ct_tcp_window_validation_query() reads the current state and doca_flow_ct_tcp_window_validation_update() replaces it:

  • Both operations must use the CT queue that allocated the entry. A wrong queue or an entry without a bound resource returns DOCA_ERROR_INVALID_VALUE. Querying while the initial profile write is pending returns DOCA_ERROR_AGAIN. With no operation pending, querying before the add or initial write has completed returns DOCA_ERROR_BAD_STATE; the same error rejects profile operations after removal starts.

  • Only one query or update may be in progress per entry. An overlapping operation, prepare rollback, or entry removal returns DOCA_ERROR_AGAIN until completion is processed. A full operation queue also returns DOCA_ERROR_AGAIN; process the queue and retry submission. A synchronous timeout or an unrecoverable operation queue returns DOCA_ERROR_TIME_OUT.

  • After a failed add, preparing the same connection or querying/updating its profile returns DOCA_ERROR_IN_PROGRESS while cleanup continues; process the queue and retry. DOCA_ERROR_BAD_STATE means the failed entry must be removed with doca_flow_ct_rm_entry(). Prepare returns the matching entry handle on either error.

  • With no callback registered through doca_flow_port_cfg_set_cb_ct_tcp_window_validation(), both APIs are synchronous. A non-NULL callback makes both APIs asynchronous for the port, and doca_flow_ct_entries_process() delivers completions on the submission queue. The no_wait parameter controls immediate submission versus batching only in asynchronous mode; synchronous calls always submit immediately.

  • An asynchronous query output buffer must remain valid until its callback runs, or until port stop if the operation never completes. Its contents are valid only after a successful completion. Update input is copied at submission time. The callback receives profile == NULL for an update and the original output pointer for a successful query.

  • An asynchronous operation remains in progress until its callback returns. Submit another operation or remove the entry only after returning. The callback's user_ctx is the pointer passed to doca_flow_ct_add_entry(), or NULL when an initial update completes before add.

Processing and Recovering Profile Operations

Profile completions use queues independent of the entry-rule queue and do not count toward max_processed_entries in doca_flow_ct_entries_process(). A processing call may report rule completions before returning a TCP window-validation processing error, not only DOCA_ERROR_AGAIN. Preserve those completed results. For DOCA_ERROR_AGAIN due to transient profile-queue contention, retry processing after the call returns.

If a profile queue becomes unrecoverable, processing returns DOCA_ERROR_TIME_OUT and its outstanding operations produce no completion callbacks. Affected entries remain busy, so individual removal and prepare rollback continue returning DOCA_ERROR_AGAIN. Stop waiting for those callbacks and destroy the CT pipe or stop the port to release the entries.

During pipe destruction, remaining profile operations are detached without delivering their callbacks, then drained with a bounded wait. Resources still referenced by a failed queue are not reused before port stop. Keep an asynchronous query buffer until its callback runs or the port stops: destroying the pipe alone does not end that buffer-lifetime requirement.

Matching the Packet Syndrome

After TCP window validation, a downstream pipe can match match.parser_meta.tcp_window_validation_syndrome. The byte contains independent state and packet-status fields:

Bits

Meaning

[7:6]

One connection state: VALID (0x00), INVALID (0x40), or DISABLED (0x80)

[5:2]

Reserved and ignored in match values and masks.

[1:0]

Combinable packet flags: BAD_PACKET (0x01) and STATE_CHANGE (0x02)

Packet flags can coexist with any connection state, so do not compare the whole byte against a state enumerator. To match the state alone, use DOCA_FLOW_CT_TCP_WINDOW_VALIDATION_PKT_SYNDROME_STATE_MASK. An explicit mask is required for VALID: its zero value without a mask leaves the field ignored rather than matching established flows.

A mask containing only reserved bits leaves the syndrome unconstrained. An all-ones pipe-template value makes the field changeable per entry. At entry creation, 0xff is a concrete value, not a template sentinel; with a full syndrome mask, it is normalized to 0xc3 by clearing reserved bits [5:2].

A fixed pipe template that constrains both state bits to 11 is rejected. A partial state mask constrains only the selected bit. Entry values, including those passed to doca_flow_pipe_control_add_entry(), are normalized without validating the state combination, so the application must select the intended state and mask.

To distinguish every documented state and packet flag in one pipe, use explicit changeable match: pipe match 0xFF, pipe mask 0xC3, and a per-entry match with the concrete syndrome value. A constant pipe match without the all-ones sentinel yields a single fixed entry.

TCP window validation reports a syndrome, not an allow/drop decision; use downstream rules to forward, drop, or send packets to software according to application policy. The syndrome is not available as an action-descriptor source or destination.

Matching the TCP window-validation syndrome together with match.parser_meta.meter_color is unsupported. Creating a pipe, ordered list, or control entry with both fields in the same match returns DOCA_ERROR_NOT_SUPPORTED.

API

For the library API reference, refer to DOCA Flow and CT API documentation in the NVIDIA DOCA Library APIs.

DOCA Flow CT is in the DOCA Flow library.

The following sections provide additional details about the library API.

enum doca_flow_ct_flags

Optional DOCA Flow CT configuration flags.

Flag

Description

DOCA_FLOW_CT_FLAG_STATS

Enables internal pipe counters for packet tracking. Call doca_flow_pipe_dump(<ct_pipe>) to dump the changed counter values.

DOCA_FLOW_CT_FLAG_WORKER_STATS

Deprecated. Retained for backward compatibility only; no longer has any effect.

DOCA_FLOW_CT_FLAG_NO_AGING

Disables aging.

DOCA_FLOW_CT_FLAG_ASYMMETRIC_TUNNEL

Allows utilizing tunnel or non-tunnel configurations in different directions.

DOCA_FLOW_CT_FLAG_NO_COUNTER

Disables counters and aging entirely to save aging-thread CPU cycles.

DOCA_FLOW_CT_FLAG_ITERATOR

Enables the entry iterator.

DOCA_FLOW_CT_FLAG_DUP_FILTER_UDP_ONLY

Applies the connection duplication filter strictly for UDP connections.

DOCA_FLOW_CT_FLAG_ORIGIN_WIRE

Indicates origin traffic will arrive from the wire. If set, mark actions can be utilized in the origin direction.

DOCA_FLOW_CT_FLAG_REPLY_WIRE

Indicates reply traffic will arrive from the wire. If set, mark actions can be utilized in the reply direction.

enum doca_flow_ct_entry_flags

Optional DOCA Flow CT entry flags.

Flag

Description

DOCA_FLOW_CT_ENTRY_FLAGS_NO_WAIT = (1 << 0)

Entry is not buffered; send to hardware immediately

DOCA_FLOW_CT_ENTRY_FLAGS_DIR_ORIGIN = (1 << 1)

Apply flags to origin direction

DOCA_FLOW_CT_ENTRY_FLAGS_DIR_REPLY = (1 << 2)

Apply flags to reply direction

DOCA_FLOW_CT_ENTRY_FLAGS_IPV6_ORIGIN = (1 << 3)

Origin direction is IPv6; origin match union in struct doca_flow_ct_match is IPv6

DOCA_FLOW_CT_ENTRY_FLAGS_IPV6_REPLY = (1 << 4)

Reply direction is IPv6; reply match union in struct doca_flow_ct_match is IPv6

DOCA_FLOW_CT_ENTRY_FLAGS_COUNTER_ORIGIN = (1 << 5)

Apply counter to origin direction

DOCA_FLOW_CT_ENTRY_FLAGS_COUNTER_REPLY = (1 << 6)

Apply counter to reply direction

DOCA_FLOW_CT_ENTRY_FLAGS_COUNTER_SHARED = (1 << 7)

Counter is shared for both direction (origin and reply)

DOCA_FLOW_CT_ENTRY_FLAGS_ENTRY_FINALIZE = (1 << 8)

Enable finalize callback on entry removed

DOCA_FLOW_CT_ENTRY_FLAGS_ALLOC_ON_MISS = (1 << 9)

Allocate on entry not found when calling doca_flow_ct_entry_prepare() API

DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_ORIGIN = (1 << 10)

Enable duplication filter on origin direction

DOCA_FLOW_CT_ENTRY_FLAGS_DUP_FILTER_REPLY = (1 << 11)

Enable duplication filter on reply direction

DOCA_FLOW_CT_ENTRY_FLAGS_STATS_UPDATES = (1 << 12)

Enable doca_flow_ct_stats_update_cb calls on counter changes

DOCA_FLOW_CT_ENTRY_FLAGS_TCP_WINDOW_VALIDATION = (1 << 13)

Bind a connection tracking offload resource

CT Configuration

CT settings moved from a separate global configuration to DOCA Flow’s standard configuration objects: per-system settings on doca_flow_cfg, per-port settings on doca_flow_port_cfg, and per-pipe settings on doca_flow_pipe_cfg.

struct doca_flow_cfg *cfg;
ret = doca_flow_cfg_create(&cfg);
doca_flow_cfg_set_ct_queues(cfg, n_queues);
doca_flow_cfg_set_ct_ctrl_queues(cfg, n_ctrl_queues);
doca_flow_cfg_set_queue_depth(cfg, queue_depth);
/* ... */
ret = doca_flow_init(cfg);

struct doca_flow_port_cfg *port_cfg;
ret = doca_flow_port_cfg_create(&port_cfg);
doca_flow_port_cfg_set_ct_flags(port_cfg, flags);
doca_flow_port_cfg_set_ct_direction(port_cfg, false, match_inner, zone_match_mask, meta_modify_mask);
/* ... */
ret = doca_flow_port_start(port_cfg, &port);

Configuration API methods:

Removed global function

Replacement function

Description

doca_flow_ct_cfg_set_queues

doca_flow_cfg_set_ct_queues

Sets the number of hardware queues utilized to manipulate connections.

doca_flow_ct_cfg_set_ctrl_queues

doca_flow_cfg_set_ct_ctrl_queues

Sets the number of CT control queues used for defining shared actions.

doca_flow_ct_cfg_set_queue_depth

doca_flow_cfg_set_queue_depth

Sets the queue depth shared by all pipe queues, including CT queues.

doca_flow_ct_cfg_set_flags

doca_flow_port_cfg_set_ct_flags

Sets the CT flags (refer to enum doca_flow_ct_flags).

doca_flow_ct_cfg_set_direction

doca_flow_port_cfg_set_ct_direction

Configures the origin and reply directions for this port.

doca_flow_ct_cfg_set_actions_mem_size

doca_flow_port_cfg_set_ct_actions_mem_size

Sets the total CT actions memory size in bytes for this port.

doca_flow_ct_cfg_set_status_update_cb

doca_flow_port_cfg_set_ct_stats_update_cb

Sets the stats update callback to notify the application of counter changes.

doca_flow_ct_cfg_set_aging_core

doca_flow_port_cfg_set_ct_aging_core

Deprecated and ignored (removal planned); use doca_flow_port_cfg_set_service_threads_core

doca_flow_ct_cfg_set_aging_query_delay

doca_flow_port_cfg_set_ct_aging_query_delay

Sets the CT aging query delay for newly created connections on this port.

doca_flow_ct_cfg_set_aging_plugin_ops

doca_flow_port_cfg_set_ct_aging_plugin_ops

Defines custom aging logic callbacks (falls back to default logic if omitted).

doca_flow_ct_cfg_set_entry_private_data_size

doca_flow_pipe_cfg_set_ct_entry_private_data_size

Sets the size of user private data allocated per connection on this pipe.

doca_flow_ct_cfg_set_entry_finalize_cb

doca_flow_pipe_cfg_set_ct_entry_finalize_cb

Sets the entry finalize callback to query final connection statistics.

Additional configuration notes:

  • CT pipe session limits and duplication filter size are configured via doca_flow_pipe_cfg during pipe creation (see CT Pipe Configuration).

  • CT counter configuration: DOCA Flow must be configured in per-port mode using doca_flow_cfg_set_resource_mode(cfg, DOCA_FLOW_RESOURCE_MODE_PORT). Define the number of CT counters via doca_flow_port_cfg_set_nr_resources(port_cfg, DOCA_FLOW_RESOURCE_COUNTER_CT, <n>).

  • TCP window-validation configuration: Set a non-zero CT queue count before DOCA Flow initialization, then define the per-port pool with DOCA_FLOW_RESOURCE_TCP_WINDOW_VALIDATION before port start. See Configuring the Resource Pool.

  • Default zone match: for a direction with no zone match mask (doca_flow_port_cfg_set_ct_direction() not called for that direction, or called with zone_match_mask = NULL), CT matches the zone on pkt_meta bits 4–31.

struct doca_flow_ct_actions

This structure is used in the following cases:

  • For defining shared actions. In this case, action data is provided by the user. The action handle is returned by DOCA Flow CT.For defining an entry with actions. The structure can be filled with two options:

    • With action handle of a previously created shared actionWith non-shared action data

DOCA Flow CT action structure.

enum doca_flow_resource_type  resource_type;
union {
               /* Used when creating an entry with a shared action. */
               uint32_t action_handle;

               /* Used when creating an entry with non-shared action or when creating a shared action. */
               struct {   
                          uint32_t action_idx;
                          struct doca_flow_meta meta;
                          struct doca_flow_header_l4_port l4_port;
                          union {
                                    struct doca_flow_ct_ip4 ip4;
                                    struct doca_flow_ct_ip6 ip6;
                           };
               } data;
       };

Where:

Field

Description

enum doca_flow_resource_type resource_type

Shared/non-shared action

uint32_t action_handle

Shared action handle

uint32_t action_idx

Actions template index

struct doca_flow_meta meta

Modify meta values

struct doca_flow_header_l4_port l4_port

UDP or TCP source and destination port

struct doca_flow_ct_ip4 ip4

Source and destination IPv4 addresses

struct doca_flow_ct_ip6 ip6

Source and destination IPv6 addresses

The value in meta, l4_port, ip4, and ip6 should start from bit0, the least significant bit, regardless of which bits are set in mask. For example, action_val.meta.u32[0] = DOCA_HTOBE32(0x12), action_mask.meta.u32[0] = DOCA_HTOBE32(0x0000FF00) sets bits 15-8 to 0x12.

DOCA Flow Connection Tracking Samples

This section describes DOCA Flow CT samples based on the DOCA Flow CT pipe.

The samples illustrate how to use the library API to manage TCP/UDP connections. 

All the DOCA samples described in this section are governed under the BSD-3 software license agreement.

Running the Samples

  1. Refer to the following documents:

  2. To build a given sample, run the following command. If you downloaded the sample from GitHub, update the path in the first line to reflect the location of the sample file: 

    cd /opt/mellanox/doca/samples/doca_flow/flow_ct_udp
    meson /tmp/build
    ninja -C /tmp/build
    

    The binary doca_flow_ct_udp is created under /tmp/build/samples/.

  3. Sample (e.g., doca_flow_ct_udp) usage: 

    Usage: doca_<sample_name> [DOCA Flags] [Program Flags]
       
    DOCA Flags:
      -h, --help                              Print a help synopsis
      -v, --version                           Print program version information    
      -l, --log-level                         Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
      --sdk-log-level                         Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
      -j, --json <path>                       Parse command line flags from an input json file
        
    Program Flags:
      -p, --pci_addr <PCI-ADDRESS>            PCIe device address
    
  4. For additional information per sample, use the -h option: 

    /tmp/build/samples/<sample_name> -h
    

The following is a CLI example for running the samples when port 08:00.0 is configured (multi-port e-switch) as manager port: 

/tmp/build/samples/doca_<sample_name> -- -r pci/08:00.0,pf0vf0 -l 60

The following is a CLI example for running the samples when port 08:00.0 is configured (multi-port e-switch) as manager port and 08:00.1is configured as the representor of the second uplink:

/tmp/build/samples/doca_<sample_name> -- -r pci/08:00.1 -l 60

To avoid the test being impacted by unexpected packets, it only accepts packets like the following examples:

  • IPv4 destination address is 1.1.1.1IPv6 destination address is 0101:0101:0101:0101:0101:0101:0101:0101

Samples List

Sample Name

Description

doca_flow_ct_2_ports

Deploys two independent e-switches, each maintaining its own distinct CT state and pipeline.

doca_flow_ct_aging

Demonstrates CT aging using a pipe with entries that feature variable aging times and custom user data.

doca_flow_ct_iterator

Iterates through the CT pipe across two standalone e-switches.

doca_flow_ct_tcp

TCP CT with TCP window validation: a port pool, a liberal default profile, and session create/destroy.

doca_flow_ct_tcp_actions

Attaches both shared and non-shared actions to a TCP CT implementation.

doca_flow_ct_tcp_entry_finalize

Leverages the CT entry finalize callback when sessions terminate or are manually removed.

doca_flow_ct_tcp_ipv4_ipv6

Handles complex flows where each packet direction utilizes a different IP version.

doca_flow_ct_tcp_stats_updates

Reports per-direction session counters through the CT stats update callback and entry private data.

doca_flow_ct_tcp_wire_to_wire

Executes a mark action on the CT for a strictly wire-to-wire TCP path.

doca_flow_ct_udp

Deploys a basic UDP pipeline that natively incorporates a CT pipe.

doca_flow_ct_udp_query

Queries the Flow CT UDP session state based on the origin or reply direction.

doca_flow_ct_udp_single_match

Creates a hardware CT entry applying a single-direction match within doca_flow_ct_add_entry().

doca_flow_ct_udp_tunnel_asymmetric

Implements an asymmetric tunnel mode for Flow CT (an extension of the core UDP query sample).

doca_flow_ct_udp_update

Dynamically updates CT entries post-creation, allowing inactive UDP sessions to receive updated aging timeouts.

Last updated: