DOCA Telemetry Diag for Windows

This guide provides instructions on building and developing applications which require collecting telemetry information provided by NVIDIA® BlueField® and NVIDIA® ConnectX® families of networking platforms.

1. Introduction

doca_telemetry_diag provides programable access to an on-device mechanism which allows sampling of diagnostic data (e.g., statistics and counters). doca_telemetry_diag allows configuring such parameters as required data IDs or sampling period, and retrieving the generated information in several formats.

1.1. Architecture

Diagnostic data is stored in firmware as a cyclic buffer of samples. Each sample represents the values of all the requested diagnostic data IDs and their corresponding sampling timestamps.

1.2. Device and Ownership

The DOCA telemetry library requires a ConnectX/BlueField DOCA device to sample from. The device can be accessed using any of its physical functions (PFs). 

The device supports two ownership models:

1. Single domain

2. Multiple domains

1.2.1. Single Domain Ownership Model

In this ownership model only one domain exists per device. Upon creation, the doca_telemetry_diag context assumes control of the associated hardware resources to prevent conflicts and ensure accurate data sampling. In rare instances, ownership may be overridden (e.g., if a process crashes before releasing ownership). 

1.2.2. Multi Domain Ownership Model

In this ownership model a device can support multiple domains. Each doca_telemetry_diag context will use one of the available domains. This allows for the case where if the first domain does not use all the hardware resources then a second domain could be created to utilize the remaining resources. There are some technical limitations when using multi domain mode, the most notable of these are:

1. Synchronized start is not supported.

2. Some Data ID values require a hardware resource that cannot be shared between domains. If another domain attempts to use a Data ID which requires the occupied hardware resource, it will be rejected

3. The force ownership flag is ignored.

4. The achieved sampling period for a context can be impacted by any already started contexts on the same device.

For WinOF-2 users, there is not any difference for a single doca_telemetry_diag instance to run under single-domain or multi-domain mode. User applications developed for single-domain mode does not need any change to run under multi-domain mode.

1.3. Configuration Phase

1.3.1. Configuration

The context can be configured to match the application use case.

To discover if a certain configuration is supported, or what the min/max value for it is, refer to section "Device Support".

1.3.1.1. Sampling Mode

The DOCA Telemetry Diagnostics library supports the following operational sampling methods:

• Single sampling – Samples are collected and stored until the firmware buffer is filled, at which point sampling is automatically terminated. 

  You can manually restart the context to collect new samples, which will overwrite the previous data in the buffer.

• Repetitive sampling – Samples are continuously collected. When the firmware buffer fills, new samples automatically overwrite the oldest samples. This process continues indefinitely until the context is explicitly stopped.

• On demand – The device does not continuously collect samples in the background. Instead, the device fetches a single sample of the data strictly upon each manual query.

1.3.1.1.1. Considerations for Repetitive Sampling Mode

When configuring the DOCA Telemetry Diagnostics library to repetitive sampling, it is important to ensure that the firmware buffer is adequately sized to handle the data flow between hardware sampling and software retrieval. The firmware buffer size is dictated by the log_max_num_samples property.

1. Determine sampling rates:

   • Hardware sampling rate – the frequency at which the hardware collects data (e.g., every 100 µsec)

   • Software retrieval rate – the average time interval between successive data retrievals by the software (e.g., every 500 msec)

Calculate AverageSamplesPerRetrieval using the following equation:

For example: 

1. To ensure smooth data processing and prevent data loss, the firmware buffer should be large enough to accommodate at least twice the average number of samples collected during the retrieval period:

   For example: 

Moreover, the number of retrieved samples should be enlarged if the retrieving process may spike occasionally. For example, if the process time between retrieval calls is up to 6 times of the average, then the number of samples should be multiplied by 6+1=7.

1.3.1.2. Sampling Period

The sampling period can be configured using doca_telemetry_diag_set_sample_period.

In some cases, depending on the number and type of data IDs configured, the actual sampling period may be higher. The actual sampling period can be queried using doca_telemetry_diag_get_sample_period after configuring the data IDs.

After starting the doca_telemetry_diag context with doca_telemetry_diag_start, the actual sampling period may differ from the desired sampling rate. To address this, you can dynamically adjust the sampling period using doca_telemetry_diag_reconfig_sample_period.

1.3.1.3. Synchronized Start

Diagnostics data is sampled by the device every given sampling period. When sampling this way, each data entry in a sample may be recorded at a slightly different time.

Synchronized start mode enables diagnostics counters to begin all data measurements at the same time (i.e., during the same clock cycle). This way, the sample period is guaranteed to be identical for all samples. Synchronized start diagnostic counters can be configured to be cleared at the beginning of each sampling period.

The following diagrams illustrate how synchronized start affects the sampling timeline:

1.3.1.4. Data IDs

The on-device mechanism provides the following diagnostic data classes:

• Counter – monotonically increasing and counting different events in the device.

  • If doca_telemetry_diag_set_data_clear is set, the counters are cleared at the beginning of each sampling period (valid only if synchronized start mode is used and operational mode is set to single or repetitive sampling).

• Statistic – other collected diagnostic data about the performance of the device. Statistic diagnostic data is cleared on each sample.

Each diagnostic data is represented by a unique identifier, the data ID. Appendix "List of Supported Data IDs" lists the currently supported data IDs.

After applying the configuration, the list of data IDs to be sampled should be applied by calling doca_telemetry_diag_apply_counters_list_by_id.

1.3.1.5. Output Formats

The doca_telemetry_diag context supports the following layout modes for sampled data:

The sample layout of these modes is illustrated in the following diagrams:

1.3.2. Device Support

DOCA Telemetry Diagnostics requires a device to operate. For picking a device, refer to "DOCA Core Device Discovery".

As device capabilities may change (see DOCA Core Device Support), it is recommended to select your device using the following method:

• doca_telemetry_diag_cap_is_supported

Some devices can allow different capabilities as follows:

• The maximum number of data IDs

• The maximum number of samples that can be held by the firmware

• Support for data clear

• Support for synchronized start

• Support for different sample modes

• Support for different timestamp sources

1.4. Execution Phase

Once the doca_telemetry_diag context is started, samples can be retrieved by calling the doca_telemetry_diag_query_counters function.

The library only retrieves new samples without duplications and returns fewer samples than requested if there are no more new samples.

There is a slight difference in behavior between the different sampling modes:

• In case of "On demand" sampling mode, each time the doca_telemetry_diag_query_counters is called, a new sample is captured and retrieved. 

• For "Single" and "Repetitive" sampling modes, multiple samples can be retrieved in a single call. The application defines the maximum number of samples it wishes to retrieve and supplies a buffer large enough to contain these samples.

• In case of "Single" sampling mode, once sampling is terminated, the user may call doca_telemetry_diag_restart to restart the sampling process, overwriting the previous results.

• During the execution phase, you can dynamically adjust the sampling period using doca_telemetry_diag_reconfig_sample_period to change the sampling rate. Note that this function has no effect in "On demand" sampling mode. For other sampling modes, it's recommended to verify the actual sampling period using doca_telemetry_diag_get_sample_period after reconfiguration to ensure it matches the expected value.

1.5. State Machine

The following section describes the different states the doca_telemetry_diag context goes through, how to move between states and what is allowed in each state.

1.5.1. Idle

The context is Idle and has ownership of a diagnostics data domain.

In this state it is expected for the application to:

• Destroy the context (releases the ownership).

• Apply a configuration, moving the context to "Configured" state.

Allowed operations:

• Configuring the context according to section Configurations.

It is possible to reach this state as follows:

1.5.2. Configured

In this state it is expected for application to:

• Apply the list of data IDs configuration using doca_telemetry_diag_apply_counters_list_by_id, moving the context to "Ready" state.

Allowed operations:

• Checking if a data ID is supported using doca_telemetry_diag_check_data_id 

• Calling stop, moving it to "Idle" state

It is possible to reach this state as follows:

1.5.3. Ready

All the necessary configuration was applied and the context is ready to start sampling.

In this state it is expected that application:

• Start the context, moving it to "Running" state.

Allowed operations:

• Calling stop, moving it to "Idle" state.

It is possible to reach this state as follows:

1.5.4. Running

In this state samples are generated and can be retrieved.

In this state it is expected that application:

• Query the counters.

Allowed operations:

• For "Single" sample mode, restarting the context if needed.

• Calling stop, moving it to "Idle" state

It is possible to reach this state as follows:

2. Prerequisites

The Windows system must meet the following baseline requirements:

2.1. WinOF-2 Package

The WinOF-2 package (containing the doca_telemetry_diag distribution) is installed or extracted

2.2. Environment

DOCA Telemetry Diag applications can run on the host machine (ConnectX-7 or BlueField-3 and above) for both x86_64 and ARM64 platform.

2.3. Acceptable Windows PCI Address Format

• Bus:Device.Function (XX:XX.X)

  • This format is identical to the Linux format

• Segment:Bus:Device.Function (XXXX:XX:XX.X)

  • The “segment“ is equivalent to the “domain“ in Linux

  • Thus this format is identical to the Linux format

• SegmentBus:Device.Function (XXXX:XX.X)

  • The value-of-SegmentBus = value-of-Segment * 256 + value-of-Bus

  • This is only for Windows

• For example, on a ARM64 Windows server, we can get the following available Mellanox devices PCI address:

> mlx5cmd -stat
the pci segment is part of the bus dword
NIC 1: physical_location=Bus 3, Device 0, Function 0
NIC 2: physical_location=Bus 515, Device 0, Function 0
NIC 3: physical_location=Bus 4099, Device 0, Function 0
NIC 4: physical_location=Bus 4611, Device 0, Function 0

• Then, users need to manually change decimal value to hex value:

• Thus, the acceptable address for the above 4 NICs are

3. DOCA Telemetry Diagnostics Sample

This section describes a telemetry diagnostics sample based on the doca_telemetry_diag library.

The sample illustrates the usage of DOCA telemetry diagnostics APIs to initialize and configure the doca_telemetry_diag context, as well as querying and parsing diagnostic counters.

3.1. Building the Sample

After the WinOF-2 package is installed or extracted, the customer can find the doca_telemetry_diagsample code directory, and all the necessary DOCA header files.

3.1.1. Sample steps to build customer own json-c library

Open a Windows command terminal:
> git clone https://github.com/Microsoft/vcpkg.git
> cd vcpkg
> bootstrap-vcpkg.sh
> bootstrap-vcpkg.bat
> vcpkg.exe integrate install
> vcpkg.exe install json-c

3.1.2. Build Sample Command

> cd $CUSTOEMR_DOCA_TELMETRY_DIAG_SAMPLE_DIRECTORY
> meson setup build -Djson_c_root=$THE_ABOVE_JSON_C_DIRECTORY
> ninja -C build

3.2. Running the Sample

3.2.1. Sample Usage

Usage: doca_telemetry_diag [DOCA Flags] [Program Flags]

DOCA Flags:
  -h, --help                        Print a help synopsis
  -v, --version                     Print program version information
  -l, --log-level                   Set the (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  --sdk-log-level                   Set the SDK (numeric) log level for the program <10=DISABLE, 20=CRITICAL, 30=ERROR, 40=WARNING, 50=INFO, 60=DEBUG, 70=TRACE>
  -j, --json <path>                 Parse command line flags from an input json file

Program Flags:
  -p, --pci-addr                    DOCA device PCI device address
  -di, --data-ids                   Path to data ids JSON file
  -o, --output                      Output CSV file - default: "/tmp/out.csv"
  -rt, --sample-run-time            Total sample run time, in seconds
  -sp, --sample-period              Sample period, in nanoseconds
  -ns, --log-num-samples            Log max number of samples
  -sr, --max-samples-per-read       Max num samples per read
  -sm, --sample-mode                sample mode (0 - single, 1 - repetitive, 2 - on demand)
  -of, --output-format              output format (0 - output_format_0, 1 - output_format_1, 2 - output_format_2)
  -sym, --sync-mode                 sync mode (0 - sync_mode_no_sync, 1 - sync_mode_sync_start)
  -tss, --timestamp-source          timestamp source (0 - timestamp_source_frc, 1 - timestamp_source_rtc)
  -f, --force-ownership             Force ownership when creating context
  -e, --example-json-path           Generate an example json file with the default data_ids to the given path and exit immediately. This file can be used as input later on. All other flags are ignored
  -rs, --reconfig-sample-period     Reconfig sample period, in nanoseconds. It means after the first sampling run with the sample_period, if the reconfig_sample_period is non-0, before the current diag instance is stopped and destroyed, the second sampling run will be done based on the reconfig_sample_period.

3.2.2. Sample Logic

1. Locating a DOCA device.

2. Initializing and configuring the doca_telemetry_diag instance.

3. Applying a list of data IDs to sample (either from a source JSON file or the default data IDs).

4. Starting the doca_telemetry_diag instance.

5. Allocating a buffer according to the sample size and amount of desired samples.

6. Querying the actual sampling period, after start.

7. Retrieving samples and writing the retrieved data to a *.csv file (either once or periodically).

8. If reconfig_sample_period is non-zero, retrieving samples and writing the sample file again based on the reconfigured sampling period.

9. Stopping the data IDs sampling.

10. Releasing all resources and destroying the context.

3.2.3. Sample Running Commands with Different PCI Address

> doca_telemetry_diag.exe -p 03:00.0
[2026-04-30 15:43:04:801018][7640][DOCA][INF][CORE][doca_log.cpp:1281][_common_write_version_to_backend] DOCA version 3.5.0011
[2026-04-30 15:43:04:802139][7640][DOCA][INF][TELEMETRY_DIAG][telemetry_diag_main.c:1012][main] Starting the sample
[2026-04-30 15:43:05:821181][7640][DOCA][INF][TELEMETRY_DIAG][telemetry_diag_main.c:1069][main] Sample finished successfully

> doca_telemetry_diag.exe -p 1203:00.0
[2026-04-30 15:43:04:801018][7640][DOCA][INF][CORE][doca_log.cpp:1281][_common_write_version_to_backend] DOCA version 3.5.0011
[2026-04-30 15:43:04:802139][7640][DOCA][INF][TELEMETRY_DIAG][telemetry_diag_main.c:1012][main] Starting the sample
[2026-04-30 15:43:05:821181][7640][DOCA][INF][TELEMETRY_DIAG][telemetry_diag_main.c:1069][main] Sample finished successfully

4. Appendix – List of Supported Data IDs

The following table lists the data IDs currently supported by DOCA:

[1] This counter includes loopback traffic and does not include packets discarded due to FCS, frame size, and similar errors.

[2] Dividing this counter by pcie_link_latency_total_read_packets yields the average PCIe read latency of those reads.

Last updated: September 01, 2026