Networking DOCA SDK Documentation

DOCA GPUNetIO

This document provides an overview and configuration instructions for DOCA GPUNetIO API. 

Introduction

The quality status of DOCA libraries is listed here.

DOCA GPUNetIO enables real-time GPU processing for network packets, making it ideal for application domains such as:

  • Signal processing

  • Network security

  • Information gathering

  • Input reconstruction

Traditional approaches often rely on a CPU-centric model, where the CPU coordinates with the NIC to receive packets in GPU memory using GPUDirect RDMA. Afterward, the CPU notifies a CUDA kernel on the GPU to process the packets. However, on low-power platforms, this CPU dependency can become a bottleneck, limiting GPU performance and increasing latency.

DOCA GPUNetIO addresses this challenge by offering a GPU-centric solution that removes the CPU from the critical path. By combining multiple NVIDIA technologies, it provides a highly efficient and scalable method for network packet processing.

Technologies integrated with DOCA GPUNetIO:

  • GPUDirect RDMA – Enables direct packet transfer between the NIC and GPU memory, eliminating unnecessary memory copies

  • GPUDirect Async Kernel-Initiated (GDAKI) – Allows CUDA kernels to control network operations without CPU intervention

    • GDAKI is also named IBGDA when used with the RDMA protocol

  • GDRCopy Library – Allows the CPU to access GPU memory directly

  • NVIDIA BlueField DMA Engine – Supports GPU-triggered memory copies

The following is an example diagram of a CPU-centric approach:

image2023-3-17_17-16-6.png

The following is an example diagram of a GPU-centric approach:

image2023-4-19_11-47-9.png

Key features of DOCA GPUNetIO include:

  • GPUDirect Async Kernel-Initiated (GDAKI)

    • GDAKI network communications – a GPU CUDA kernel can control network communications to send or receive data

      • GPU can control Ethernet communications (Ethernet/IP/UDP/TCP/ICMP)

      • GPU can control RDMA communications (InfiniBand or RoCE are supported)

      • CPU intervention is unnecessary in the application critical path

  • GPUDirect RDMA

    • Enables direct data transfers to and from GPU memory without CPU staging copies.

  • DMA Engine Control

    • CUDA kernels can initiate memory copies using BlueField's DMA engine

  • Semaphores for Low-Latency Communication

    • Supports efficient message passing between CUDA kernels or between CUDA kernels and CPU threads

  • Smart Memory Allocation

    • Allocates aligned GPU memory buffers, optimizing memory access

    • GDRCopy library to allocate a GPU memory buffer accessible from the CPU

  • Accurate Send Scheduling

    • Provides precise control over Ethernet packet transmission based on user-defined timestamps.

NVIDIA applications that use DOCA GPUNetIO include:

  • Aerial 5G SDK – For ultra-low latency 5G network operations

  • NIXL – NVIDIA Inference Xfer Library (NIXL) is targeted for accelerating point to point communications in AI inference frameworks (e.g., NVIDIA Dynamo)

  • NVQLink – New GPU RoCE transceiver Holoscan sensor bridge operator

  • UCX – new GDAKI module via GPUNetIO functions

  • NCCL – GIN transport enabled via GPUNetIO GPU communications

  • NVSHMEM – GPUNetIO transport for GPU communications replacing the IBGDA transport

  • DeepEP/Hybrid/EP

For more information about DOCA GPUNetIO, refer to the following NVIDIA blog posts:

Changes From Previous Releases

Changes in 3.6.0

  • GPUNetIO changes/fixes:

    • New capability probes:

      • doca_gpu_nic_cap_is_gpu_mem_umem_supported()

      • doca_gpu_nic_cap_is_host_mem_umem_supported()

      • doca_gpu_nic_cap_is_nic_handler_gpu_sm_db_supported()

    • BlueFlame ring_bf() on sm_90+ now fences the TMA copy with fence.proxy.async and commits and waits on it.

    • BlueFlame submit_bf_warp() now also rings the regular doorbell and honors its sync scope.

    • doca_gpu_destroy() now returns DOCA_ERROR_INVALID_VALUE ("mtable map is not empty") when allocations were never freed

    • doca_gpu_mem_alloc()/free() are now thread-safe on a shared doca_gpu: a mutex protects the allocation table

    • New doca_gpu_dev_verbs_signal_warp(): one FetchAdd per lane, submitted once

    • put_signal_warp and p_signal_warp (and get_warp with MCST) now return the warp's last WQE as out_ticket in every lane

    • Verbs and Ethernet lock/unlock (CTA/GPU modes) are rewritten in inline PTX to work around a CUDA 13.4 compiler issue

    • doca_gpu_dev_{verbs,eth}_atomic_add() now use atomicAdd()/atomicAdd_block() by default instead of cuda::atomic_ref (24d768f1e69).

      • DOCA_GPUNETIO_{VERBS,ETH}_USE_CCC_ATOMIC_ADD=1 restores the 3.5 behaviour.

      • With the default, 8-bit and 16-bit types now fail a static_assert.

    • mark_wqes_ready() on an RQ in GPU sharing mode uses the lighter NVIDIA-NIC acquire fence, CTA scope by default

    • Improved CQE attribute filling in doca_gpu_dev_eth_rxq_recv_warp and doca_gpu_dev_eth_rxq_recv_block

  • GPUNetIO Ethernet device headers

    • Rx fix:

      • recv_warp and recv_block now exit per warp or block with __any_sync/__syncthreads_or, replacing a block-wide shared flag.

      • An error CQE no longer makes one thread break out alone while the others wait at a barrier, which could hang.

      • recv_block now handles 2-D and 3-D blocks.

    • Per-packet attributes (bytes, timestamp) no longer have holes. The new DOCA_GPUNETIO_ETH_RX_ATTR_ENFORCE_CORRECTNESS (default 1) fills them for [0, out_pkt_num) after the polling loop.

    • DOCA_GPUNETIO_ETH_ENABLE_DEBUG is renamed DOCA_GPUNETIO_ETH_ENABLE_PRINT_DEBUG with no alias, so the old define silently stops working.

    • New doca_gpu_dev_eth_warp_sum() and an opt-in DOCA_GPUNETIO_ETH_RX_USE_WARP_SUM (default 0).

  • GPUNetIO Verbs samples improvements:

    • New -z/--data-direct in 7 verbs samples (not get_bw or twosided_bw) (2a95fe6d593).

      • It needs CUDA ≥ 12.8 and the right libmlx5 headers.

      • With -z 1, the bandwidth servers receive into GPU memory.

      • Verbs samples now need libmlx5 to build.

    • Breaking:

      • Peers exchange their config (-i, -t, -x) and abort on a mismatch, so 3.5 and 3.6 peers can't be mixed.

      • Options are validated at startup (2e0147b1172).

      • -n 3 (BlueFlame) is rejected everywhere except write_lat

      • The --nic-handler long option now works; it was "nic handler" with a space.

  • GPUNetIO Ethernet samples improvements:

    • gpunetio_simple_send :

      • Sends valid Eth/IPv4/UDP packets, with new --src-ip/--dst-ip.

      • -t is optional (default 512) and checked more strictly; -s must be 42–9216.

      • Send errors now fail the sample, and teardown happens in the right order.

    • gpunetio_simple_receive: block-wide exit and error handling, DOCA Flow pipes destroyed before the RXQ, and leaks fixed.

    • gpunetio_send_wait_time:

      • All 8 bursts are queued up front. In 3.5 it waited for each burst before posting the next, so with a short -t later bursts went out late.

      • Native wait-on-time support is now checked before the TXQ is created, with an mlxconfig hint (ConnectX-7+).

      • New failure: a NIC-vs-host clock offset above 25 ms now stops the sample with DOCA_ERROR_BAD_STATE.

    • Size DOCA Flow counters per port (PORT resource mode) instead of the deprecated global call

  • Data direct enabled in DOCA Mmap: data direct can be combined with GPUNetIO Ethernet applications

  • GPUNetIO Ethernet Application improved:

    • New UDP options:

      • --udp-rx-scope thread|warp|block

      • --udp-rx-max-pkts, --udp-rx-timeout

      • --udp-rx-pkt-num, --udp-rx-pkt-size

      • --udp-packet-filtering, --udp-stage-latency, --udp-cqe-attr-check

      • --udp-pkt-host-mem, --udp-rq-host-mem

    • Behaviour change:

      • Each UDP receive call now takes at most 512 packets with a 500 µs timeout; 3.5 had no limit and a 1 ms timeout.

      • DNS classification only runs with --udp-packet-filtering.

      • The [UDP] line reports bandwidth per 5 s interval and adds TOTAL_PPS, so 3.5 output parsers break.

    • UDP kernel fixes: block-uniform exit, block-wide error handling, 64-bit stats atomics, and __threadfence_system() before the semaphore is set READY.

General Performance and Best Practices

Adhere to the following guidelines to maximize performance and streamline development with DOCA GPUNetIO.

Build Configuration

  • Use release builds: Always compile applications and samples by setting buildtype = 'release' in the meson.build file. This ensures maximum throughput and minimal latency compared to the default debug mode.

Hardware and System Setup

  • Driver Selection for dmabuf: Install the nvidia-open drivers to utilize dmabuf. If the environment relies on the closed-source NVIDIA drivers (cuda-drivers), the system does not support dmabuf; use nvidia-peermem instead.

  • Optimize PCIe topology: For peak throughput, ensure the GPU and NIC share a PIX or PXB topology (connected via a single PCIe bridge). Avoid NODE or SYS topologies, as routing data across NUMA nodes or SMP interconnects reduces efficiency.

  • Expand BAR1 size: If an application fails to map memory buffers to the NIC, verify the GPU's BAR1 size using nvidia-smi -q. Enable Resizable BAR in the system BIOS to allocate sufficient space.

  • Systems Without GPUDirect RDMA: For hardware that does not support GPUDirect RDMA (such as DGX Spark), use the DOCA_GPU_MEM_TYPE_CPU_GPU memory type and enable CPU proxy mode for transmissions.

Programming Guidelines

  • CUDA context initialization: Ensure an active CUDA context exists on the target device before calling DOCA GPUNetIO functions (such as doca_gpu_create). Initialize this context effectively by calling cudaFree(0).

  • Optimize execution scopes: When utilizing high-level Ethernet APIs, select the widest possible execution scope (Warp or Block) rather than Thread scope. This minimizes contention on atomic operations and significantly increases doorbell ringing efficiency.

  • Strict pointer management: Strictly separate memory access boundaries to prevent segmentation faults. Use memptr_gpu exclusively within CUDA kernels, and use memptr_cpu solely for CPU-side management.

Permissions

  • Root requirements: Execute all Ethernet samples and applications with sudo or root privileges, as they rely on DOCA Flow.

  • Non-root alternatives: Verbs, RDMA, and DMA operations can run without root access. To enable this functionality, configure the NVIDIA driver with the option NVreg_RegistryDwords="PeerMappingOverride=1;".

Choose the path that matches your current goal:

Goal

Description

Link

Set up my system

Install packages, configure NIC firmware (ConnectX/BlueField), and verify PCIe topology

Installation and Setup

Learn the concepts

Understand GDAKI, CPU vs. GPU control paths, and the GPUNetIO memory model

Architecture and Design

Start coding

Reference CPU and GPU functions for Ethernet, RDMA, Verbs, and DMA

API Reference

Run a demo

Walk through build instructions and execute provided sample applications

Sample Guide

Last updated: