DOCA Management

This guide provides instructions on how to use the DOCA Management API for developing applications for device management and configuration.

Introduction

DOCA Management provides an API to manage NVIDIA® BlueField® DPU and host devices and their associated functions (PFs, VFs, and SFs).

Using DOCA Management, administrators and applications can dynamically configure and query device settings, enable or disable capabilities, and manage device behavior at runtime through a standardized API that communicates directly with device firmware via the Linux kernel's fwctl framework.

This document is intended for software developers and system administrators wishing to manage and configure DPU and host devices programmatically, control device capabilities, and integrate device management into their applications.

Prerequisites

DOCA Management library uses the Linux kernel’s fwctl framework and as such requires the following kernel drivers to be available and loaded:

• fwctl.ko – fwctl device firmware access framework

• mlx5_fwctl.ko – mlx5 ConnectX fwctl driver

The library also uses DOCA Core Device for device and device representor discovery. It is recommended to read the following section before proceeding:

• DOCA Core Device

Library Changes From Previous Releases

Changes in 3.3.0

Added

• doca_mgmt_icm_quota.h

Changes in 3.4.0

Added

• doca_mgmt_diagnostics_data.h

• doca_mgmt.h:

  • doca_mgmt_dev_rep_ctx_create_by_pci_addr()

Changes in 3.5.0

Added

• doca_mgmt_raw_common.h

• doca_mgmt_raw_register.h

• doca_mgmt_raw_hca_cap.h

• doca_mgmt.h (management target discovery and target-string context creation APIs):

  • doca_mgmt_target_info_dev_create_list() / doca_mgmt_target_info_dev_destroy_list()

  • doca_mgmt_target_info_dev_rep_create_list() / doca_mgmt_target_info_dev_rep_destroy_list()

  • doca_mgmt_target_info_dev_get_*() and doca_mgmt_target_info_dev_rep_get_*() accessors

  • doca_mgmt_dev_ctx_create_from_target_string()

  • doca_mgmt_dev_rep_ctx_create_from_target_string()

Environment

DOCA Management-based applications can run on the host machine and on the NVIDIA® BlueField® DPU.

Management operations are performed either through a management device context or through a management device representor context. The choice of context depends on the operation: some operations are performed on the device itself, while others (such as configuring the capabilities of a child VF/SF) can only be performed through the function’s representor. Representor-based operations must therefore be executed where the representor resides (for example, on the BlueField® DPU or on the host PF that owns the representor, depending on the deployment).

Since the library communicates with the device firmware through the Linux kernel’s fwctl framework, the invoking process must have sufficient permissions to access the corresponding fwctl character device.

Certain operations impose additional requirements on the target function. For example, setting device capabilities requires the represented device to be uninitialized (i.e., unbound from the mlx5_core driver) at the time the capabilities are set.

Architecture

The library operates through management device contexts and management device representor contexts. These management contexts are the main objects of the library. They hold a management interface for the device and are passed to other APIs of the library to specify the device to operate on.

To operate on a device or representor, the application must first identify it. This can be done either through the library’s own management target discovery, which enumerates the manageable devices and representors present on the system (see Management Contexts and Targets), or through DOCA Core device discovery.

The library provides two sets of APIs:

• Typed, feature-specific APIs – each targets a specific feature and exposes a dedicated, strongly typed interface for it (for example, doca_mgmt_icm_quota, doca_mgmt_cc_global_status, etc.). These APIs are built around dedicated management handles (see Dedicated Management Handles) that encapsulate the feature’s configuration attributes and provide setters/getters together with query/modify operations that apply or read the attributes to or from the device. This is the recommended way to configure a feature, if a dedicated API for it exists.

• Generic APIs (the doca_mgmt_raw_* family) – provide low-level access to the device for advanced users. They allow querying and modifying device registers and HCA capabilities by name, as well as sending arbitrary firmware commands directly to the device. These APIs are intended for cases where a dedicated typed API is not available, and require detailed knowledge of the underlying device interface (see Raw APIs).

Objects

Management Contexts and Targets

The DOCA Management device context (doca_mgmt_dev_ctx) represents a management interface to a device and is used to perform device-level management operations. Similarly, the DOCA Management device representor context (doca_mgmt_dev_rep_ctx) represents a management interface to a device representor and is used to perform management operations for the device represented by the representor.

The management device context and management device representor context function as handles that are passed to other DOCA Management APIs to specify the target device. Some DOCA Management APIs accept a management device context, while others accept a management device representor context.

A management target is a device or device representor that can be managed by the library. The library provides a discovery mechanism that enumerates the management targets present on the system, so that an application can identify and select the device or representor to operate on.

The discovery APIs return opaque target information objects – doca_mgmt_target_info_dev for devices and doca_mgmt_target_info_dev_rep for device representors. Each object exposes accessors for the target’s attributes (such as its PCI function type, interface name, and, for representors, the controller, PF, VF/SF indices, and VHCA ID) and, in particular, a target string that uniquely identifies the target.

A management context is obtained by passing a target string, discovered through the discovery APIs, to the corresponding target-string context creation API (see Device Context Creation). The target string formats are:

Alternatively, a management context can be created directly from a DOCA Core device or device representor, without going through management target discovery (see DOCA Core Device).

Dedicated Management Handles

The library provides dedicated handles for specific management operations. Each handle is an opaque structure that encapsulates configuration attributes and provides APIs to both configure the handle (via setters/getters) and apply/query these attributes to or from the device.

The following table lists the currently supported handles:

Raw API Objects

Raw API allows configuring device registers and HCA capabilities. Unlike the typed feature-specific APIs, the generic doca_mgmt_raw_* APIs are not built around dedicated handles. Instead, they reference registers and capability types by name and their contents are expressed as a list of fields.

The library provides APIs to enumerate the supported registers and HCA capability types, and to describe the fields of a given register or HCA capability type. The describe APIs return field descriptors, each represented by a doca_mgmt_raw_field_desc structure, which holds:

• name – the name of the field.

• bit_size – the size of the field, in bits.

• access – the access type of the field, one of the doca_mgmt_raw_field_access enum values:

  • DOCA_MGMT_RAW_FIELD_ACCESS_RO – read-only.

  • DOCA_MGMT_RAW_FIELD_ACCESS_WO – write-only.

  • DOCA_MGMT_RAW_FIELD_ACCESS_RW – read-write.

  • DOCA_MGMT_RAW_FIELD_ACCESS_INDEX – an index field, provided as input to select the specific entity to access.

  • DOCA_MGMT_RAW_FIELD_ACCESS_OP – an operation field, provided as input to select the operation to perform.

  • DOCA_MGMT_RAW_FIELD_ACCESS_UNKNOWN – the access type is not known.

The library also provides APIs to query and modify the fields of a register or HCA capability type by name. A field value is represented by a doca_mgmt_raw_field structure, which holds:

• name – the name of the field.

• value – the 32-bit value of the field.

For HCA capabilities, a query or modify additionally specifies a capability mode (applicable only for query) and a target function. The capability mode is given by the doca_mgmt_raw_hca_cap_mode enum, which selects which set of values is accessed:

• DOCA_MGMT_RAW_HCA_CAP_MODE_MAX – the maximum capability values supported by the device.

• DOCA_MGMT_RAW_HCA_CAP_MODE_CUR – the current, effective capability values.

The target function is given by the doca_mgmt_raw_hca_cap_target structure, which identifies the function whose capabilities are accessed, according to its type:

• DOCA_MGMT_RAW_HCA_CAP_TARGET_TYPE_SELF – the function executing the operation.

• DOCA_MGMT_RAW_HCA_CAP_TARGET_TYPE_OTHER_FUNCTION_ID – another function, identified by its function ID.

• DOCA_MGMT_RAW_HCA_CAP_TARGET_TYPE_OTHER_VHCA_ID – another function, identified by its VHCA ID.

Finally, the raw command API allows sending an arbitrary firmware command directly to the device. The command is described by an input and output payload and the appropriate command scope. The scope of the command is given by thedoca_mgmt_cmd_scope enum which maps exactly to the fwctl command scopes:

• DOCA_MGMT_CMD_SCOPE_CONFIGURATION

• DOCA_MGMT_CMD_SCOPE_DEBUG_READ_ONLY

• DOCA_MGMT_CMD_SCOPE_DEBUG_WRITE

• DOCA_MGMT_CMD_SCOPE_DEBUG_WRITE_FULL

See Raw APIs for the expected usage flow.

Expected Usage Flow

The following section describes the expected usage flow of the DOCA Management library. It first covers the creation of the management contexts, which is common to all operations, and then presents representative examples: ICM quota configuration as an example of a typed, feature-specific API, and the raw APIs as an example of the generic APIs. The remaining typed feature-specific APIs follow the same pattern as the ICM quota example; refer to the Dedicated Management Handles table and the DOCA Management Samples for details on each of them.

Device Context Creation

To start using the library, a DOCA Management device context (doca_mgmt_dev_ctx) must be created. The context can be created in one of two ways: from a management target string, or from a DOCA device.

Creating a Device Context From a Target String

1. Obtain a device target string:

   1. Create the list of device targets by calling doca_mgmt_target_info_dev_create_list().

   2. Select the desired device by inspecting the attributes of the returned entries using the accessors, and retrieve its target string using doca_mgmt_target_info_dev_get_target_string().

   3. Free the list using doca_mgmt_target_info_dev_destroy_list() when it is no longer needed. The target string is owned by its list entry, so copy it if it must outlive the list.

2. Create the doca_mgmt_dev_ctx by calling doca_mgmt_dev_ctx_create_from_target_string() with the target string.

3. The created doca_mgmt_dev_ctx can now be used to configure the device or to create a DOCA Management device representor context for one of its representors (as explained in Device Representor Context Creation). When the application finishes using it, destroy it using doca_mgmt_dev_ctx_destroy().

Creating a Device Context From a DOCA Device

1. Open the doca_dev that should be configured.
   For information about opening a DOCA device, see DOCA Core Device Discovery.

2. Create the corresponding doca_mgmt_dev_ctx by calling doca_mgmt_dev_ctx_create() with the opened doca_dev.

3. The created doca_mgmt_dev_ctx can now be used to configure the device or to create a DOCA Management device representor context for one of its representors (as explained in Device Representor Context Creation).

4. When the application finishes using the doca_mgmt_dev_ctx, destroy it using doca_mgmt_dev_ctx_destroy(). The doca_dev must remain valid until the doca_mgmt_dev_ctx is destroyed.

Device Representor Context Creation

Some configurations (such as configuring device capabilities) can be done only via the device representor. For this, a DOCA Management device representor context (doca_mgmt_dev_rep_ctx) must be created. As with the device context, it can be created in one of two ways: from a management target string, or from a DOCA device representor.

Creating a Device Representor Context From a Target String

1. Obtain a representor target string:

   1. Create the list of device targets by calling doca_mgmt_target_info_dev_create_list() and select the representor’s parent device, as described for a device target string above.

   2. Create the list of that device’s representor targets by calling doca_mgmt_target_info_dev_rep_create_list(), passing the doca_mgmt_target_info_dev of the desired parent device.

   3. Select the desired representor by inspecting the attributes of the returned entries using the accessors, and retrieve its target string using doca_mgmt_target_info_dev_rep_get_target_string().

   4. Free the lists using doca_mgmt_target_info_dev_rep_destroy_list() and doca_mgmt_target_info_dev_destroy_list() when they are no longer needed. The target string is owned by its list entry, so copy it if it must outlive the list.

2. Create the doca_mgmt_dev_rep_ctx by calling doca_mgmt_dev_rep_ctx_create_from_target_string() with the representor target string. The library resolves the target string internally and creates the underlying management device context on the application’s behalf.

3. Use the created doca_mgmt_dev_rep_ctx to configure the represented device, and destroy it using doca_mgmt_dev_rep_ctx_destroy() when finished. In this case the representor management context owns the underlying management device context and destroys it automatically.

Creating a Device Representor Context From a DOCA Device Representor

1. Create a doca_mgmt_dev_ctx for the parent device of the representor (as explained in Device Context Creation). For example, if VF0 of PF0 should be configured, create a doca_mgmt_dev_ctx for PF0.

2. Open the doca_dev_rep that should be configured. For information about opening a DOCA device representor, see DOCA Core Device Representor Discovery.

3. Create the corresponding doca_mgmt_dev_rep_ctx by calling doca_mgmt_dev_rep_ctx_create() with the created doca_mgmt_dev_ctx and the opened doca_dev_rep.

4. The created doca_mgmt_dev_rep_ctx can now be used to configure the device represented by the representor. 

5. When the application finishes using the doca_mgmt_dev_rep_ctx, destroy it using doca_mgmt_dev_rep_ctx_destroy(). Both the doca_dev_rep and doca_mgmt_dev_ctx must remain valid until the doca_mgmt_dev_rep_ctx is destroyed.

ICM Quota Configuration

ICM Quota configuration is supported on both devices and representors.

Querying ICM Quota Capabilities

Not all devices support ICM quota configuration. To check for support and discover capabilities:

1. Create a doca_mgmt_dev_ctx for the target device.

2. Check for support by calling doca_mgmt_cap_icm_quota_is_supported().

3. If supported, retrieve the maximum quota limit using doca_mgmt_cap_icm_quota_get_max_limit().  

   This returned value represents the maximum valid input for doca_mgmt_icm_quota_set_limit().

   If a device supports ICM Quota, its associated representors also support it and share the same limit.

Getting ICM Quota Configuration

To retrieve the current configuration:

1. Create a doca_mgmt_dev_ctx or a doca_mgmt_dev_rep_ctx for the device or device representor that should be configured (as explained in Device Context Creation and Device Representor Context Creation).

2. Initialize a doca_mgmt_icm_quota handle for the device or the device representor by calling doca_mgmt_icm_quota_create_for_dev() or doca_mgmt_icm_quota_create_for_dev_rep() with the created doca_mgmt_dev_ctx or doca_mgmt_dev_rep_ctx, respectively. The created handle will be associated with the given device or device representor and the following query will be applied to this device or device representor.

3. Query the ICM quota configuration by calling doca_mgmt_icm_quota_query().

4. Use the following getters to read the results:

   • doca_mgmt_icm_quota_get_limit()

   • doca_mgmt_icm_quota_get_current_allocation()

   • doca_mgmt_icm_quota_get_max_reached()

5. Destroy the handle using doca_mgmt_icm_quota_destroy().

Setting ICM Quota Configuration

1. Create a doca_mgmt_dev_ctx or a doca_mgmt_dev_rep_ctx for the device or device representor that should be configured (as explained in Device Context Creation and Device Representor Context Creation).

2. Initialize a doca_mgmt_icm_quota handle for the device or the device representor by calling doca_mgmt_icm_quota_create_for_dev() or doca_mgmt_icm_quota_create_for_dev_rep() with the created doca_mgmt_dev_ctx or doca_mgmt_dev_rep_ctx, respectively. The created handle will be associated with the given device or device representor and the following configuration will be applied to this device or device representor.

3. Set the relevant attributes to the handle:

   1. Set the quota limit using doca_mgmt_icm_quota_set_limit().

   2. Reset the "max reached" counter using doca_mgmt_icm_quota_set_reset_max_reached().

4. Commit the configuration by calling doca_mgmt_icm_quota_modify().

5. Destroy the handle using doca_mgmt_icm_quota_destroy().

Raw APIs

For advanced use cases, the library provides the generic doca_mgmt_raw_* APIs, which allow accessing the device directly rather than through a dedicated, feature-specific abstraction. These APIs are organized into three groups: raw register access, raw HCA capability access, and raw firmware commands.

All raw APIs operate on a management device context (doca_mgmt_dev_ctx), created as explained in Device Context Creation.

Raw Register

The raw register API allows querying and modifying device registers by name. A register is expressed as a set of named fields.

1. Create a doca_mgmt_dev_ctx for the target device.

2. Optionally, retrieve the list of supported registers by calling doca_mgmt_raw_register_list(). The returned list must be freed using doca_mgmt_raw_str_list_free().

3. Optionally, retrieve the field descriptors of a specific register by calling doca_mgmt_raw_register_describe() with the register name. The returned list must be freed using doca_mgmt_raw_field_desc_list_free().

4. Query a register:

   • To query specific fields, call doca_mgmt_raw_register_query() with the register name, optional input fields, and the output fields to read.

   • To query all fields, call doca_mgmt_raw_register_query_all()with the register name and optional input fields. The returned fields must be freed using doca_mgmt_raw_field_list_free().

5. To modify a register, call doca_mgmt_raw_register_modify() with the register name and the input fields to set.

Raw HCA Capability

The raw HCA capability API allows querying and modifying the HCA capabilities of a function by capability type name, following the same field-based model as the raw register API.

1. Create a doca_mgmt_dev_ctx for the target device.

2. Optionally, retrieve the list of supported HCA capability types by calling doca_mgmt_raw_hca_cap_list(). The returned list must be freed using doca_mgmt_raw_str_list_free().

3. Optionally, retrieve the field descriptors of a specific HCA capability type by calling doca_mgmt_raw_hca_cap_describe() with the capability type name. The returned list must be freed using doca_mgmt_raw_field_desc_list_free().

4. Query HCA capabilities:

   • To query specific fields, call doca_mgmt_raw_hca_cap_query() with the capability type name, the capability mode (max or current), the target function (the function that executes the operation, or another function identified by its function ID or VHCA ID), and the output fields to read.

   • To query all fields, call doca_mgmt_raw_hca_cap_query_all() with the capability type name, the capability mode (max or current), the target function (the function that executes the operation, or another function identified by its function ID or VHCA ID). The returned fields must be freed using doca_mgmt_raw_field_list_free().

5. To modify HCA capabilities, call doca_mgmt_raw_hca_cap_modify() with the target function and the input fields to set. The target function must be a function other than the one executing the operation (identified by its function ID or VHCA ID).

Raw Command

The raw command API allows sending an arbitrary firmware command directly to the device. The following steps detail how to send a raw command to a device:

1. Create a doca_mgmt_dev_ctx for the device that should receive the command.

2. Prepare the input payload buffer according to the specific command requirements.

3. Allocate an output payload buffer to receive the command response.

4. Call doca_mgmt_raw_cmd() with the following parameters:

   • The created doca_mgmt_dev_ctx

   • A command identifier string (for logging)

   • The command scope

   • Pointer to the input payload buffer and its size

   • Pointer to the output payload buffer and its size

5. Check the return value to verify that the command executed successfully.

6. Parse the output payload buffer according to the specific command response format.

The command scope parameter specifies the scope of the command being performed and must match the fwctl scope assigned to it by the mlx5_fwctl driver.

The mapping between DOCA and fwctl command scopes is as follows:

DOCA Management Samples

This section describes DOCA Management samples based on the DOCA Management library. Each sample is a standalone command-line tool. The samples demonstrate how to use the DOCA Management API for the following device management tasks:

• Configure device data direct capability.

• Configure device congestion control global status.

• Configure and query ICM quota.

• Configure and query diagnostics data.

• Access device registers, HCA capabilities, and firmware commands directly through the raw APIs.

Running the Samples

Refer to the following documents:

• DOCA Installation Guide for Linux  for details on how to install BlueField-related software.

• NVIDIA BlueField Platform Software Troubleshooting Guide for any issue you may encounter with the installation, compilation, or execution of DOCA samples.

To build a given sample, run the following command. If you downloaded the sample from GitHub, update the path in the first line to reflect the location of the sample file:

cd /opt/mellanox/doca/samples/doca_mgmt/<sample_name>
meson /tmp/build
ninja -C /tmp/build

Sample usage (e.g., doca_mgmt_data_direct):

Usage: doca_mgmt_data_direct [Program Commands] [DOCA Flags]

Program Commands:
  get                               Get data direct capability
  list-targets                      List the available management targets
  set                               Set data direct capability

DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse command line flags from an input json file

For additional information per sample, use the -h option:

/tmp/build/doca_<sample_name> -h
/tmp/build/doca_<sample_name> get -h
/tmp/build/doca_<sample_name> list-targets -h
/tmp/build/doca_<sample_name> set -h

Most samples follow a common command pattern:

• list-targets — enumerate the management targets available on the system (supported by all samples).

• caps — check whether the capability is supported by the target (where applicable).

• get — read the current configuration.

• set — modify the configuration.

The exact set of commands and parameters varies per sample (for example, mgmt_raw groups its operations under the register and hca-cap sub-commands). See the per-sample sections below for each sample’s specific commands, parameters, and examples.

Listing Management Targets

All the samples share a common list-targets command that enumerates the management targets available on the system and prints their target strings. A target string printed by list-targets can be passed directly to a sample’s -d/--device parameter (device target) or -r/--rep parameter (representor target).

$ doca_mgmt_<sample> list-targets [-v/--verbose]

By default only the target strings are printed. Pass -v/--verbose to additionally print each target’s attributes. In samples that manage representors, each device’s representor targets are printed nested beneath it.

The device and representor target string formats are described in Management Contexts and Targets.

Examples

• List the device targets (target strings only):

  Bash

  $ doca_mgmt_diagnostics_data list-targets
  pci/0000:08:00.0
  pci/0000:08:00.1
  

• List the device targets and their representors, with all attributes:

  Bash

  $ doca_mgmt_data_direct list-targets --verbose
  pci/0000:08:00.0
      pci_func_type       PF
      native_port_num     1
      iface_name          eth2
      ibdev_name          mlx5_0
      pci/0000:08:00.0,c0pf0vf0
          pci_func_type       VF
          controller_num      0
          pf_index            0
          vf_index            0
          vhca_id             2
          iface_name          eth4
  

Samples

Management Data Direct

The NVIDIA® ConnectX®-8 device may expose a side DMA engine as an additional PCIe PF called the Data Direct device. This additional device allows access to data buffers through multiple PCIe data path interfaces. By default, the data direct capability is disabled for VFs and SFs and must be explicitly enabled to allow data direct usage for a specific VF or SF.

This sample demonstrates how to set and get the data direct capability for VFs and SFs using the DOCA Management library.

The sample supports three commands:

• list-targets – Lists the available management targets (see Listing Management Targets).

• get – Displays the current data direct capability state.

• set – Enables or disables the data direct capability.

For getand setcommands, the device (VF or SF) to operate on must be specified.

Specifying the Device

• For VFs:

  1. Specify the VF representor using the -r or --rep parameter with a VF representor target string. For example:

     pci/0000:08:00.0,c0pf0vf0
     

  2. Specify the VF PCIe address (e.g., 0000:08:00.2) using the -v or --vf-pci-addr parameter.

• For SFs, specify the SF representor using the -r or --rep parameter with a SF representor target string. For example:

  pci/0000:08:00.0,c0pf0sf88
  

Set Command

The -e or --enabled parameter must be specified with true or false values to indicate whether the data direct capability should be enabled or disabled, respectively.

Example Executions

• Get data direct state for a VF with PCIe address 0000:08:00.2:

  Bash

  $ /tmp/build/doca_mgmt_data_direct get --vf-pci-addr 0000:08:00.2
  

• Enable data direct for a VF (uninitialized):

  Bash

  $ /tmp/build/doca_mgmt_data_direct set --vf-pci-addr 0000:08:00.2 --enabled true
  

• Disable data direct for a VF (uninitialized):

  Bash

  $ /tmp/build/doca_mgmt_data_direct set --vf-pci-addr 0000:08:00.2 --enabled false
  

• Enable data direct using VF representor:

  Bash

  $ /tmp/build/doca_mgmt_data_direct set --rep pci/0000:08:00.0,c0pf0vf0 --enabled true
  

• Enable data direct for SF representor:

  Bash

  $ /tmp/build/doca_mgmt_data_direct set --rep pci/0000:08:00.0,c0pf0sf88 --enabled true
  

• List the available targets:

  Bash

  $ /tmp/build/doca_mgmt_data_direct list-targets
  

Sample Logic

• Get Data Direct:

  1. Creating a DOCA management device representor context for the target VF/SF: either from the representor target string or from the VF PCI address.

  2. Creating a device caps general handle.

  3. Executing get operation through the DOCA management device caps general API.

  4. Retrieving the data direct capability for the specified representor interface.

  5. Cleaning up all DOCA management and device structures.

• Set Data Direct:

  1. Creating a DOCA management device representor context for the target VF/SF: either from the representor target string or from the VF PCI address.

  2. Creating a device caps general handle and setting data direct attribute.

  3. Executing set operation through the DOCA management device caps general API.

  4. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/mgmt_data_direct_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/mgmt_data_direct_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_data_direct/meson.build

Management Congestion Control Global Status

This sample demonstrates how to set and get congestion control global status for different priorities and protocols (NP/RP) on a DOCA device using the DOCA Management library.

The sample supports three commands:

• list-targets – Lists the available management targets (see Listing Management Targets).

• get – Shows the current congestion control global status.

• set – Enables or disables the congestion control global status.

For get and set, the device to operate on must be specified using the -d/--device parameter with a device target string. The following parameters are also required:

• -p/--priority – The congestion control priority (0–7).

• -t/--protocol – The congestion control protocol, rp or np.

• -e/--enabled – For set, specified with true or false to enable or disable the congestion control, respectively.

Example Executions

• List the available targets:

  Bash

  $ /tmp/build/doca_mgmt_cc_global_status list-targets
  

• Get the congestion control global status for a device, priority 3, protocol np:

  Bash

  $ /tmp/build/doca_mgmt_cc_global_status get --device pci/0000:08:00.0 --priority 3 --protocol np
  

• Enable the congestion control global status for a device, priority 0, protocol rp:

  Bash

  $ /tmp/build/doca_mgmt_cc_global_status set --device pci/0000:08:00.0 --priority 0 --protocol rp --enabled true
  

The sample logic includes two main operations:

• Get Congestion Control Global Status:

  1. Creating a DOCA management device context from the target string that was specified in the command line arguments.

  2. Creating a congestion control global status handle and setting priority and protocol attributes.

  3. Executing get operation through the DOCA management congestion control global status API.

  4. Retrieving the congestion control global status enabled attribute for the specified priority and protocol.

  5. Cleaning up all DOCA management and device structures.

• Set Congestion Control Global Status:

  1. Creating a DOCA management device context from the target string that was specified in the command line arguments

  2. Creating a congestion control global status handle and setting priority, protocol and enabled attributes.

  3. Executing set operation through the DOCA management congestion control global status API.

  4. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/mgmt_cc_global_status_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/mgmt_cc_global_status_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_cc_global_status/meson.build

Management ICM Quota

This sample allows getting, setting, and querying capabilities for ICM quota on a given device or device representor (PF/VF/SF), and demonstrates how to do it using DOCA management library API.

The sample supports four commands:

• list-targets – Lists the available management targets (see Listing Management Targets).

• caps – Shows the ICM quota capabilities for the device.

• get – Shows the current ICM quota configuration (limit, current allocation, max reached allocation).

• set – Sets the ICM quota limit or resets the max reached allocation counter.

For caps, get, and set, either a device or a device representor must be specified. The -d/--device and -r/--rep parameters are mutually exclusive.

Specifying the Device/Device Representor

• Specify a device using the -d or --device parameter with a device target string. Example for a device with PCIe address 0000:08:00.0:

  pci/0000:08:00.0
  

• Specify a device representor using the -r or --rep parameter with a representor target string:
  

  1. Example for a VF:

     pci/0000:08:00.0,c0pf0vf0
     

  2. Example for a SF:
     

     pci/0000:08:00.0,c0pf0sf88
     

Caps Command

Shows whether the device supports ICM quota and, if so, also shows the maximum ICM quota limit that can be set by the set command. The capabilities reported for a device also apply to its representors.

Get Command

The following optional parameters can be specified to retrieve specific attributes:

• --limit shows the ICM quota limit that is configured for the device.

• --cur-alloc shows the currently allocated ICM quota of the device.

• --max-reached shows the maximum reached ICM quota that the device has reached so far.

If no parameters are specified, all attributes are retrieved.

Set Command

At least one of the following parameters must be specified:

• -L or --limit sets the ICM quota limit for the device (e.g., 4096, 4K, 1M, 1G, 1T, unlimited). The value must be aligned to 4K. The value must be less than or equal to the maximum ICM quota limit reported by caps command and less than or equal to 16TB-8KB. Value of 'unlimited' indicates no limit.

• --reset-max-reached resets the maximum reached ICM quota counter.

Example Executions

• Get ICM quota capabilities for a device:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota caps --device pci/0000:08:00.0
  

• Get all ICM quota attributes for a device:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota get --device pci/0000:08:00.0
  

• Get only the ICM quota limit for a device:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota get --device pci/0000:08:00.0 --limit
  

• Set ICM quota limit to 1GB for a VF:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota set --rep pci/0000:08:00.0,c0pf0vf0 --limit 1G
  

• Set ICM quota limit to unlimited for a device:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota set --device pci/0000:08:00.0 --limit unlimited
  

• Reset the ICM quota max reached counter for a SF:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota set --rep pci/0000:08:00.0,c0pf0sf88 --reset-max-reached
  

• List the available targets:

  Bash

  $ /tmp/build/doca_mgmt_icm_quota list-targets
  

Sample Logic

• Get ICM Quota:

  1. Creating a DOCA management device context from the device target string, or a DOCA management device representor context from the representor target string.

  2. Creating an ICM quota handle.

  3. Executing get operation through the DOCA management ICM quota API.

  4. Retrieving and displaying the requested attributes (limit, current allocation, max reached).

  5. Cleaning up all DOCA management and device structures.

• Set ICM Quota:

  1. Creating a DOCA management device context from the device target string, or a DOCA management device representor context from the representor target string.

  2. Creating an ICM quota handle and setting the requested attributes (limit, reset max reached).

  3. Executing set operation through the DOCA management ICM quota API.

  4. Cleaning up all DOCA management and device structures.

• Get ICM Quota Capabilities:

  1. Creating a DOCA management device context from the device target string.

  2. Checking if ICM quota is supported on the device.

  3. Retrieving and displaying the maximum ICM quota limit supported by the device.

  4. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_icm_quota/mgmt_icm_quota_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_icm_quota/mgmt_icm_quota_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_icm_quota/meson.build

Management Diagnostics Data

This sample allows you to get, set, and query capabilities for diagnostics data on a given device (PF), and demonstrates how to execute this using the DOCA Management API.

The sample supports four commands (a device must be specified for caps, get, and set):

• list-targets – Lists the available management targets (see Listing Management Targets).

• caps – Shows the diagnostics data capabilities for the device.

• get – Shows the current diagnostics data configuration (multi_domain).

• set – Sets the diagnostics data multi_domain configuration.

Specifying the Device

Specify a device using the -d or --device parameter with a device target string. Example for a device with PCIe address 0000:08:00.0:

pci/0000:08:00.0

For set, the --multi-domain parameter must also be specified with true or false to enable or disable multi-domain mode, respectively.

Example Executions

• Get diagnostics data capabilities for a device:

  $ /tmp/build/doca_mgmt_diagnostics_data caps --device pci/0000:08:00.0
  

• Get diagnostics data multi_domain setting for a device:

  $ /tmp/build/doca_mgmt_diagnostics_data get --device pci/0000:08:00.0
  

• Enable multi-domain for a device:

  Bash

  $ /tmp/build/doca_mgmt_diagnostics_data set --device pci/0000:08:00.0 --multi-domain true
  

• Disable multi-domain for a device:

  Bash

  $ /tmp/build/doca_mgmt_diagnostics_data set --device pci/0000:08:00.0 --multi-domain false
  

• List the available targets:

  Bash

  $ /tmp/build/doca_mgmt_diagnostics_data list-targets
  

Sample Logic

• Get diagnostics data multi_domain:

  1. Creating a DOCA management device context from the target string.

  2. Creating a diagnostics data handle.

  3. Executing get operation through the DOCA management diagnostics data API.

  4. Retrieving and displaying the multi_domain attribute.

  5. Cleaning up all DOCA management and device structures.

• Set diagnostics data multi_domain:

  1. Creating a DOCA management device context from the target string.

  2. Creating a diagnostics data handle and setting the multi_domain attribute.

  3. Executing set operation through the DOCA management diagnostics data API.

  4. Cleaning up all DOCA management and device structures.

• Get diagnostics data capabilities:

  1. Creating a DOCA management device context from the target string.

  2. Checking if diagnostics data is supported on the device.

  3. Cleaning up all DOCA management and device structures.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_diagnostics_data/mgmt_diagnostics_data_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_diagnostics_data/mgmt_diagnostics_data_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_diagnostics_data/meson.build

Management Raw

This sample demonstrates how to use the DOCA Management raw APIs, which provide a low-level interface for device configuration intended for advanced users.

The sample illustrates how to use the raw register API to list the supported registers of a device, describe the fields of a register, query a register, and modify a register. It also illustrates how to use the raw HCA capability API to list the supported HCA capability types of a device, describe the fields of an HCA capability type, query HCA capabilities, and modify HCA capabilities.

In addition to the list-targets command (see Listing Management Targets), the sample provides two commands, register and hca-cap, each with four sub-commands. For all commands, the device must be specified using the -d/--device parameter with a device target string (e.g., pci/0000:08:00.0).

Raw Register

The register command has four sub-commands:

• list – List the available registers for the device.

• describe – List the available fields for a given register.

• get – Query a register. Returns all fields if output fields are not specified, or only the requested fields.

• set – Modify a register with the given input fields.

For describe, get, and set, the register name must be specified using -r/--register-name. For get, optional input fields (-i/--input-fields, in the format field_name1=value1,field_name2=value2,...) and output fields (-o/--output-fields, in the format field_name1,field_name2,...) can be specified. For set, input fields (-i/--input-fields) are required.

Examples

• List the available registers of a device:

  Bash

  $ doca_mgmt_raw register list --device pci/0000:08:00.0
  

• Describe the fields of the VHCA_ICM_CTRL register:

  Bash

  $ doca_mgmt_raw register describe --device pci/0000:08:00.0 --register-name VHCA_ICM_CTRL
  

• Get specific fields of the VHCA_ICM_CTRL register with input parameters:

  Bash

  $ doca_mgmt_raw register get --device pci/0000:08:00.0 --register-name VHCA_ICM_CTRL --input-fields vhca_id=2,vhca_id_valid=1 --output-fields cur_alloc_icm
  

• Set fields of the VHCA_ICM_CTRL register:

  Bash

  $ doca_mgmt_raw register set --device pci/0000:08:00.0 --register-name VHCA_ICM_CTRL --input-fields max_alloc_icm_th_mask=1,max_alloc_icm_th=0xffffffff
  

Raw HCA Cap

The hca-cap command has four sub-commands:

• list – List the available HCA capability types for the device.

• describe – List the available fields for a given HCA capability type.

• get – Query HCA capabilities. Returns all fields if output fields are not specified, or only the requested fields.

• set – Modify HCA capabilities with the given input fields.

For describe, get, and set, the HCA capability type must be specified using -c/--cap-type. For get, the capability mode -m/--cap-mode (max or current) is required, and the target function may optionally be selected using -f/--function-id or -v/--vhca-id (mutually exclusive; if neither is set, the target is the device specified by -d/--device). For set, the target function (-f/--function-id or -v/--vhca-id) and input fields (-i/--input-fields) are required. The VHCA ID of a function can be retrieved using the list-targetscommand (see Listing Management Targets).

Examples

• List the available HCA capability types of a device:

  Bash

  $ doca_mgmt_raw hca-cap list --device pci/0000:08:00.0
  

• Get all fields of the cmd_hca_cap HCA capability type in max capability mode:

  Bash

  $ doca_mgmt_raw hca-cap get --device pci/0000:08:00.0 --cap-type cmd_hca_cap --cap-mode max
  

• Get specific fields of the cmd_hca_cap HCA capability type of another function specified by its VHCA ID:

  Bash

  $ doca_mgmt_raw hca-cap get --device pci/0000:08:00.0 --cap-type cmd_hca_cap --cap-mode max --vhca-id 2 --output-fields vhca_id,log_max_qp
  

• Set fields of the cmd_hca_cap HCA capability type of another function specified by its VHCA ID:

  Bash

  $ doca_mgmt_raw hca-cap set --device pci/0000:08:00.0 --cap-type cmd_hca_cap --vhca-id 2 --input-fields selective_repeat=0x0
  

Sample Logic

• List Registers Operation:

  1. Creating a DOCA management device context from the target string.

  2. Retrieving the list of supported register names via the DOCA management raw register API.

  3. Displaying the register names and cleaning up.

• Describe Register Operation:

  1. Creating a DOCA management device context from the target string.

  2. Retrieving the list of field descriptors for the specified register via the DOCA management raw register API.

  3. Displaying the fields and cleaning up.

• Get Register Operation:

  1. Creating a DOCA management device context from the target string.

  2. Parsing optional input fields and optional output fields from the command line.

  3. If output fields were specified, querying only those fields; otherwise querying all fields via the DOCA management raw register API.

  4. Displaying the results and cleaning up.

• Set Register Operation:

  1. Creating a DOCA management device context from the target string.

  2. Parsing input fields from the command line.

  3. Modifying the register with the given fields via the DOCA management raw register API.

  4. Cleaning up.

• List HCA Capability Types Operation:

  1. Creating a DOCA management device context from the target string.

  2. Retrieving the list of supported HCA capability types via the DOCA management raw HCA cap API.

  3. Displaying the HCA capability types and cleaning up.

• Describe HCA Capability Operation:

  1. Creating a DOCA management device context from the target string.

  2. Retrieving the list of field descriptors for the specified HCA capability type via the DOCA management raw HCA cap API.

  3. Displaying the fields and cleaning up.

• Get HCA Capability Operation:

  1. Creating a DOCA management device context from the target string.

  2. Parsing optional output fields from the command line.

  3. If output fields were specified, querying only those fields; otherwise querying all fields via the DOCA management raw HCA cap API.

  4. Displaying the results and cleaning up.

• Set HCA Capability Operation:

  1. Creating a DOCA management device context from the target string.

  2. Parsing input fields from the command line.

  3. Modifying the HCA capabilities with the given fields via the DOCA management raw HCA cap API.

  4. Cleaning up.

Reference:

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_raw/mgmt_raw_sample.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_raw/mgmt_raw_main.c

• /opt/mellanox/doca/samples/doca_mgmt/mgmt_raw/meson.build

Last updated: September 01, 2026