Networking DOCA SDK Documentation

DOCA Argus Service Guide

This page provides installation, configuration, and usage instructions for the DOCA Argus service.

Introduction

DOCA Argus is a service that runs on an NVIDIA® BlueField® DPU and provides real-time situational awareness of the compute node that the DPU is attached to. It uses live machine introspection, performed at the hardware level, to analyze selected regions of volatile host memory, which is the authoritative source for the operational state of the system. DOCA Argus copies these regions with zero-copy data access techniques, without affecting application or AI workload performance.

DOCA Argus runs in a BlueField trust domain that is isolated from the host. It requires no agents, integration, or host resources, and it continues to operate even if the host is compromised. A single BlueField DPU running DOCA Argus can monitor an entire compute node in bare-metal, virtualized, containerized, sandboxed, and multi-tenant environments.

Line drawing of an NVIDIA BlueField-4 card next to four icons. No agents and no integration, runs independently of the host. Isolated from the host, operates from a separate trust domain and is invisible to attackers. No overhead, enhances system efficiency with 100 percent AI resource availability. Resilient security, trusted even when host resources are under attack.

Operation

DOCA Argus identifies the Linux kernel version running on the compute node and applies a matching kernel-specific memory map to locate the memory regions to monitor. It supports x86 and Arm64 CPUs. Using NVIDIA DOCA DMA, it reads those regions and decodes the raw data into information about processes, threads, execution states, and system behavior.

A policy engine filters out irrelevant activity and continuously analyzes memory for state changes and anomalous behavior. Activity that matches one or more policy rules is reported as an event or an alert. For definitions, see Message Type.

DOCA Argus logs events, alerts, and system activity messages locally in JSON and syslog formats. It can also export them through Fluent Bit or Vector to security platforms, data lakes, and other big data platforms, such as extended detection and response (XDR) platforms, for additional logic and enrichment. For settings, see Output and Log Types.

DOCA Argus does not extract personally identifiable information (PII). It reports only the information described in this guide, and you cannot customize the reported information.

Diagram of DOCA Argus operation. On BlueField, DOCA Argus reads the memory of an AI workload on the host through real-time memory introspection. It collects processes, threads, memory, libraries, file descriptors, network connections, Bash commands, GPU telemetry, infrastructure mapping, and container, VM, and bare-metal context. The resulting events and alerts go to local storage and local logic on BlueField, and to a partner platform for detection, data lake storage, and visualization.

Use Cases

DOCA Argus identifies the workloads running on the compute node, how they are deployed, and how they operate, starting at the process level. Security and observability platforms use this data in the following ways.

Runtime Workload Threat Detection

Containers created from the same image are expected to behave consistently, so you can create a behavioral profile for each container and monitor for deviations that might indicate compromise. To support this, DOCA Argus reports the following:

  • Executed binaries, loaded libraries, and their SHA-256 hashes

  • Execution paths and command-line arguments

  • Per-process file access, including which files are accessed, by which processes, and what actions are performed

  • Inbound and outbound network connections for each process

  • Indicators of compromise, such as executed Bash commands, reverse shell activity, and unauthorized process execution

Cybersecurity independent software vendors (ISVs) and security teams can ingest this data, together with events and alerts, into security information and event management (SIEM), security orchestration, automation, and response (SOAR), and XDR platforms. There, they can combine it with their own threat intelligence, analytics, and correlation logic to detect threats in real time and respond to them, for example, by containing a threat or preventing lateral movement.

Diagram of real-time threat detection. DOCA Argus on BlueField reads the memory of an AI workload in a Pod and collects memory, Bash commands, file descriptors, network connections, library and binary hashes, process properties, and IP addressing. Alerts and events go to a local AI and ML module, which sends detections to a partner XDR platform and triggers response actions through an AI security gateway (reverse proxy). The XDR platform can also request response actions.

AI Discovery

DOCA Argus provides the data to identify and map AI infrastructure and workloads in real time:

  • How each workload is deployed (in a container, in a virtual machine, or on bare metal) and its relationships, such as container-to-Pod, container-to-VM, and container-to-operating-system mappings

  • Container image hashes, which can be correlated with public AI repositories and internal software repositories to identify AI software, AI models, large language models (LLMs), and AI agents

  • Process-level network connections and exposed network services

This data also supports passive vulnerability assessment, runtime integrity monitoring, behavioral analysis, and AI exposure management, such as analyzing network traffic to and from an AI workload for traces of PII.

Diagram of AI discovery. DOCA Argus on BlueField reads the memory of an AI workload in a Pod and collects container-to-Pod mapping, container image hashes and names, library and binary hashes, process properties, network connections, IP addressing, container-to-container communications, OS properties, and listening ports. Events go to local storage and to a partner XDR platform, which pulls information from AI software repositories such as NVIDIA NGC, GitHub, and Hugging Face.

Observability

DOCA Argus reports how workloads interact with memory, threads, the network, the file system, CPUs, and GPUs. It maps each process in an AI workload to the AI node that runs it and to the GPUs that the process uses. For GPUs, it reports the following:

  • Physical GPU attributes, including make, model, and serial number

  • Mappings of workloads to physical or virtual GPUs

  • Process-to-GPU utilization metrics and other GPU observability data

Diagram of what DOCA Argus monitors, centered on a process. The process has parent, child, inject, scheduled, and terminated relationships. It creates, injects, and terminates threads, starts and ends network connections, creates, deletes, and modifies files, and loads and unloads libraries. It uses memory, CPU, and GPU. DOCA Argus also monitors runtime manifest violations, reverse shell use, and Bash shell use across containerized, virtualized, and bare-metal environments.

Requirements

  • Operates only on BlueField-3 or later configured in DPU mode (see BlueField Modes of Operation).

  • Requires firmware version 24.35.0388 or later.

  • Supported BlueField image versions: 4.11.0 or later.

  • The DOCA Argus service container must run in privileged mode to enable full-system DMA reads.

Limitations

  • Tested against KVM-based hypervisors only.

  • Supports Linux-based OSs (bare-metal, virtualization, containers).

  • Kata Containers are supported only if NVIDIA-DPU support is enabled.

Prerequisites

  1. Configure BlueField firmware. On BlueField, configure the base address register (BAR) of the physical function (PF):

    dpu> mlxconfig -d /dev/mst/<mst_device> s PF_BAR2_SIZE=2 PF_BAR2_ENABLE=1
    

    Replace <mst_device> with mt41692_pciconf0 for BlueField-3.

    If you monitor virtual functions (VFs), also enable single root I/O virtualization (SR-IOV) and set the number of VFs:

    dpu> mlxconfig -d /dev/mst/<mst_device> s SRIOV_EN=1 NUM_OF_VFS=<vf-number>
    
  2. Enable input-output memory management unit (IOMMU) passthrough only if it is not already enabled.

    1. Edit the GRUB configuration:

      host> sudo vim /etc/default/grub
      
    2. Add the following value to GRUB_CMDLINE_LINUX_DEFAULT:

      iommu=pt <intel/amd>_iommu=on
      
    3. Apply the changes:

      • Ubuntu:

        sudo update-grub
        
      • CentOS or RHEL:

        sudo grub2-mkconfig -o /boot/grub2/grub.cfg
        
    4. Reboot the host.

Generate Profiles Manually

DOCA Argus automatically generates the required symbol and memory-region configuration at runtime. If automatic generation fails, prepare the target system manually:

  1. Install the required debug packages on the target system:

    • Ubuntu:

      sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse
      deb http://ddebs.ubuntu.com/ $(lsb_release -cs)-proposed main restricted universe multiverse
      EOF
      sudo apt install ubuntu-dbgsym-keyring
      sudo apt-get update
      sudo apt-get install linux-image-$(uname -r)-dbgsym
      
    • CentOS or RHEL:

      sudo yum install --enablerepo=base-debuginfo \
        kernel-devel-$(uname -r) \
        kernel-debuginfo-$(uname -r) \
        kernel-debuginfo-common-$(uname -m)-$(uname -r)
      
  2. Make doca_apsh_config.py available on the target system using one of the following methods:

    • If DOCA is installed on the target system, use /opt/mellanox/doca/tools/doca_apsh_config.py and continue to the next step.

    • Otherwise, copy the script from BlueField:

      scp <bf-user>@<bf-ip>:/opt/mellanox/doca/tools/doca_apsh_config.py /tmp/
      
  3. Download a dwarf2json release binary from volatilityfoundation/dwarf2json.

  4. Create the JSON files:

    cd /opt/mellanox/doca/tools/
    pip3 install psutil pdbparse
    python3 doca_apsh_config.py --files memregions symbols --os <windows/linux> --path <path-to-dwarf2json>
    cp /opt/mellanox/doca/tools/*.* <shared-folder>
    dpu> scp <shared-folder>/* <path-to-doca-argus-binary>
    

Service Deployment

Service Configuration

DOCA Argus configuration is managed through SERVICE_CONFIG_FILE in the container YAML.

Service

  • Immediate shutdown – Terminates immediately on SIGINT or SIGTERM and skips graceful shutdown.

  • Service log level – Sets DOCA logging verbosity. The default is 50 (INFO). Supported values are 10 (DISABLE), 20 (CRITICAL), 30 (ERROR), 40 (WARNING), 50 (INFO), 60 (DEBUG), and 70 (TRACE).

  • System scanner sleep time – Sets the delay between scans. Supported units are s, m, and ms.

DOCA Argus Configuration

  • Auto scan – Scans all available systems unless the systems section is defined.

  • Profile generation – Automatically generates the required kernel data at runtime when no prebuilt data is available. Initialization can take longer when profile generation is required.

  • Default – Defines configuration values that apply unless overridden under systems.

  • Systems – Defines the monitored systems and their overrides.

Per-system Configurations

  • Representor ID – Specifies the VU ID of the VF or PF to track.

    • PF:

      host> lspci -vv -s <PF_pci_address> | grep VU | cut -d " " -f 4
      
    • VF – Append VF<x> to the PF VU ID. For example: MT2333XZ06YAMLNXS0D0F0VF1.

  • Memory regions path – Specifies a JSON file path, or auto, for the host OS memory map.

  • OS symbol path – Specifies a JSON file path or directory, or auto.

  • OS type – Specifies Linux or Windows.

  • DMA device name – Matches the representor ID. List devices with:

    dpu> ibv_devinfo | grep 'hca_id' | awk '{print $2}'
    
  • Service log level – Overrides the service logging verbosity.

  • SDK log level – Sets SDK logging verbosity.

  • Limits – Sets maximum values for string length, processes, file descriptors, threads, virtual memory areas, and kernel modules.

  • Events – Contains the following settings:

    • Container filter – Includes or excludes containerized processes.

    • Container enrichment – Enriches messages from containerized processes with container and Kubernetes Pod metadata. Enabled by default.

    • SBOM – Lists the SHA signatures of approved executables and libraries (software bill of materials).

Collection

  • Events – Enables or disables each event type.

Output

  • Log events to stdout – Enables logging to standard output.

  • Log folder path – Specifies the directory for file logs.

  • Log threshold size – Rotates logs when they reach this size.

  • Log max files count – Sets the maximum number of rotated log files.

  • Telemetry address – Specifies the aggregator address.

  • Telemetry tag – Specifies the tag used by Fluent Bit.

  • Telemetry format – Specifies JSON or syslog.

  • Telemetry user data – Specifies custom metadata to include in telemetry records.

Log Types

DOCA Argus produces the following types of logs and output.

Standard Output

DOCA Argus displays important service messages on standard output, including version information, successful startups, and startup failures.

Debug Log Output

The debug log provides complete debug output, including partial event data, trace logs, and collection failures. These logs are stored in /var/log/doca_argus/.

Event Log Output

DOCA Argus stores the complete event log in JSON format in the configured log folder. For local storage, Linux logrotate manages log rotation. You can override the default configuration in /etc/cron.d/logrotate and /etc/logrotate.d/argus.

Telemetry Output

DOCA Argus can produce telemetry records in JSON or syslog format.

Telemetry is disabled by default. To enable it, set telemetry_address in the service configuration and ensure that telemetry_tag matches the tag in the telemetry pipeline configuration.

Fluent Bit

The DOCA Argus container page on NGC provides YAML configuration examples for Fluent Bit and Vector. Use the applicable YAML file as the starting point for your telemetry deployment.

Fluent Bit runs independently from the DOCA Argus service. To run Fluent Bit locally on the DPU, use an input configuration similar to the following:

[INPUT]
    Name                  tcp
    Tag                   <your_preferred_tag>
    Listen                0.0.0.0
    Port                  24224
    Format                json

For Splunk, add the following encapsulation filter:

[FILTER]
    Name                  nest
    Match                 *
    Operation             nest
    Wildcard              *
    Nest_under            event

The following example forwards telemetry to Elasticsearch:

[INPUT]
    Name                  tcp
    Tag                   elastic_forward_input
    Listen                0.0.0.0
    Port                  24224
    Format                json

[SERVICE]
    Log_Level             info

[OUTPUT]
    Name                  es
    Match                 *
    Host                  <elasticsearch_ip>
    Port                  <elasticsearch_port>
    Index                 argus
    Suppress_Type_Name    On
    Log_Level             info

Run Fluent Bit with the configuration:

docker run --rm --net=host -v <path_to_fluentbit_conf_file>:/fluent-bit/etc/fluent-bit.conf --name fluent_bit -it fluent/fluent-bit

For additional output plugins, see the Fluent Bit manual.

Vector

Vector is an observability data pipeline that can receive DOCA Argus telemetry over TCP, decode JSON records, and forward them to supported destinations such as Elasticsearch. Vector can run independently from the DOCA Argus service or as a sidecar container in the same Kubernetes Pod.

Set telemetry_address to the Vector listener address, such as 127.0.0.1:24224, and set telemetry_format to json. The following configuration receives JSON telemetry and forwards it to Elasticsearch:

YAML
data_dir: /vector-data-dir

sources:
  tcp_json_in:
    type: socket
    mode: tcp
    address: "127.0.0.1:24224"
    decoding:
      codec: json

sinks:
  elasticsearch_out:
    type: elasticsearch
    inputs: ["tcp_json_in"]
    endpoints:
      - "http://<elasticsearch_ip>:<elasticsearch_port>"
    bulk:
      index: "argus"

Start Vector with the configuration file:

vector --config /etc/vector/vector.yaml

For Kubernetes deployments, Vector can run alongside the DOCA Argus container and share a configuration volume. DOCA Argus sends telemetry to 127.0.0.1:24224, and Vector listens on the same Pod network and forwards records to the configured sink.

YAML
containers:
  - name: vector
    image: timberio/vector:0.43.1-alpine
    imagePullPolicy: IfNotPresent
    args: ["--config", "/etc/vector/vector.yaml"]
    volumeMounts:
      - mountPath: /etc/vector
        name: conf-vector-config

Ensure that the Vector configuration volume is populated before the Vector container starts. For additional source, transform, and sink options, see the Vector documentation.

Troubleshooting

For issues with installing, deploying, or running DOCA Argus, see the DOCA SDK page of the NVIDIA BlueField Platform Software Troubleshooting Guide.

Reported Activities

This section is a reference to the activities that DOCA Argus reports. It is intended for security ISVs, system integrators (SIs), NVIDIA Cloud Partners (NCPs), and anyone who integrates with DOCA Argus.

Naming Conventions

Reported messages use the following naming conventions:

  • Parameter names use lowercase letters.

  • Names are explicit.

  • An underscore (_) separates words.

  • Parameter names for measurements include the unit of measure, for example, process_hash_sha256 or average_packet_size_bytes.

  • Enum values use uppercase letters (UPPER_CASE) and begin with a letter.

  • Path values use Linux forward slashes instead of backslashes to avoid escaping, for example, /Users/askywalker.

  • Timestamps use the ISO 8601 format YYYY-MM-DDTHH:MM:SS.nnnnnnnnn±hh:mm. On Linux-based systems, time is represented in UTC. The format has the following parts:

    • YYYY-MM-DD is the date.

    • T separates the date and the time.

    • HH:MM:SS is the time.

    • nnnnnnnnn is the nanoseconds.

    • ±hh:mm is the time zone offset from UTC, for example, +02:00 or -01:00.

  • Empty fields have the value null.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

In the current DOCA Argus version, a field with no information might appear as an empty string (""), null, or N/A.

Message Type

DOCA Argus collects raw activities from host memory and uses them to describe the operational state of a workload. The policy engine processes raw activities, filters out irrelevant content, and reports only meaningful data. Its output supports uses such as real-time visibility, telemetry, threat detection, breach detection, and indicators of attack.

A raw activity that matches one or more policy rules becomes a meaningful activity. DOCA Argus reports the following message types:

  • Event – One or more meaningful activities that show the current recorded state and provide situational awareness.

  • Alert – One or more meaningful activities that show an immediate threat or immediate impact, and that must be investigated or responded to immediately.

  • System activity – Information about the operational state of the DOCA Argus service.

Message Structure

Each reported activity uses the structure described in this section.

Schema

The following table describes the message parameters:

Parameter

Type

Parent Object

Description

message_header

object

—

Root object containing the message metadata.

vendor_name

enum

message_header

Set to NVIDIA.

product_name

enum

message_header

Set to DOCA_ARGUS.

product_version

string

message_header

The current version of the product.

message_type

enum

message_header

One of EVENT, ALERT, or SYSTEM_ACTIVITY.

severity

enum

message_header

One of INFO, ERROR, WARNING, MEDIUM, HIGH, or CRITICAL.

schema_version

string

message_header

The current version of the JSON schema.

message_id

string

message_header

The unique message identifier.

occurred_message_time_iso_8601_ns

string

message_header

BlueField card time.

message_timezone

string

message_header

BlueField card time.

user_data

string

message_header

Custom user data added to every message.

bluefield_system_information

object

message_header

Information about the BlueField system.

bluefield_networking_interfaces

array

bluefield_system_information

A list of all configured BlueField interfaces, with their names, IP addresses, and MAC addresses. Includes both IPv4 and IPv6 addresses.

bluefield_network_interface_name

string

networking_interfaces

The interface name.

bluefield_network_interface_mac_address

string

networking_interfaces

The interface MAC address.

bluefield_network_interface_ipv4_address

string

networking_interfaces

The list of IPv4 addresses associated with the interface.

bluefield_network_interface_ipv6_address

string

networking_interfaces

The list of IPv6 addresses associated with the interface.

workload_information

object

message_header

Information specific to the workload.

unique_identifier

string

workload_information

The unique identifier of the workload, if any.

os_version

string

workload_information

The OS version of the workload, for example, Linux version 5.4.0-216-generic.

boot_uuid

string

workload_information

Unique identifier for the current kernel boot.

hostname

string

workload_information

The hostname of the workload.

workload_networking_interfaces

array

workload_information

A list of all workload interfaces, with their names, IP addresses, and MAC addresses.

workload_network_interface_name

string

networking_interfaces

The interface name.

workload_network_interface_mac_address

string

networking_interfaces

The interface MAC address.

workload_network_interface_ipv4_address

string or array

networking_interfaces

The IPv4 addresses associated with the interface.

workload_network_interface_ipv6_address

string or array

networking_interfaces

The IPv6 addresses associated with the interface.

container_context

object

workload_information

Container information. Populated only if the reported workload is a container. Added in DOCA Argus 3.5.

container_id

string

container_context

The container's unique identifier.

container_name

string

container_context

The container name.

image_name

string

container_context

The container image name.

image_digest

string

container_context

The SHA-256 hash that uniquely identifies a specific build of a container image.

image_manifest_digest

string

container_context

A unique cryptographic identifier (SHA-256 hash) that represents the content of a Docker image.

pod_name

string

container_context

The name of the Pod that the container is part of.

pod_uid

string

container_context

The unique identifier of the Pod that the container is part of.

pod_ip_address

string

container_context

The IP address of the Pod that the container is part of.

enrichment_status

enum

container_context

complete or partial. Indicates whether the information collection process completed.

activity_data

object

message_header

Details about the reported activity.

name

string

activity_data

The name of the alert, event, or system activity.

<details>

object

activity_data

Detailed information for the message type.

The Pod parameters (pod_name, pod_uid, and pod_ip_address) are populated only if the workload is a container that runs in a Pod.

Example JSON Message

The following example shows the data that DOCA Argus produces for each event and alert:

JSON
{
    "vendor_name": "NVIDIA"
    "product_name": "DOCA_ARGUS"
    "product_version": "<version">
    "message_type": <"EVENT" | "ALERT" | "SYSTEM_ACTIVITY">,
    "severity": <"INFO" | "ERROR" | "WARNING" | "MEDIUM" | "HIGH" | "CRITICAL">,
    "schema_version": "1.0",
    "message_id": <"unique_message_id">,
    "occurred_message_time_iso_8601_ns": "2024-10-31T17:20:10.02.123456789+02:00",
    "message_timezone": "UTC",
    "user_data": "<STRING>",
    "bluefield_system_information": {
        "bluefield_networking_interfaces": {
            "0": {
                "bluefield_network_interface_name": "docker0",
                "bluefield_network_interface_mac_address": "00:1a:ca:ff:ff:01",
                "bluefield_network_interface_ipv4_address": "172.17.0.1",
                "bluefield_network_interface_ipv6_address": "fe80::21a:caff:feff:ff01",
            },
            "1": {
                …
            }
            …
        },
    },
    "workload_information": {
        "unique_identifier": "<>",
        "os_version": "Linux version 5.4.0-26-generic",
        "boot_uuid": "6d516e23-6fa0-45ca-907a-5cac46cd5737",
        "hostname": "5c94642421ce",
        "workload_networking_interfaces": {
            "0": {
                "workload_network_interface_name": "eth0",
                "workload_network_interface_mac_address": "52:54:00:47:5d:ad",
                "workload_network_interface_ipv4_address": "192.168.121.81",
                "workload_network_interface_ipv6_address": "fe80::5054:ff:fe47:5dad",
            },
            "1": {
                …
            }
            …
        },
    },

    "container_context": {
        "container_id": "c0798b9e9d8fdc21a96b16bf022bbcc7613e42560a307b283bd2d9953eb1b937",
        "container_name": "<NAME>",
        "image_name": "ubuntu:22.04",
        "image_digest": "sha256:714d1a2f39e9830a7c0453dc924ddf849be74811c13e4afaa21c0b9b6ad74af0",
        "image_manifest_digest": "sha256:71343a2f45e9830a7c0453dc924ddf849be74811c13e4afaa21c0b9b6ad74af0",
        "pod_name": <NAME>,
        "pod_uid": <UID>,
        "pod_ip_address": <IP_ADRESS>,
        "enrichment_status": "<complete|partial>"
        }
    },
},

<"activity_data">: {
    <"name">: <"the name of the event | alert | system_activity">,
    <<Activity data details, per event | alert | system_activity type>>
}
}

Alerts, Events, and System Activity Messages

This section lists the messages that DOCA Argus supports. The Attributes column lists the attribute objects that each message includes. For the fields in each object, see Attributes.

Events and Alerts

The following tables list the events and alerts that DOCA Argus reports, grouped by category.

Bash Shell

Type

Severity

Activity Name

Attributes

Description

Event

Info

Shell Command Executed

Process, Bash shell command

One or more new Bash shell commands were executed.

Alert

High

Shell History Cleared

Process, Bash shell command

The Bash shell history was cleared.

Alert

High

Shell History Disabled

Process, Bash shell command

The Bash shell history was disabled.

Containers

Type

Severity

Activity Name

Attributes

Description

Event

Info

Container Started

Process

A new container instance was detected.

Event

Info

Container Terminated

Process

A container was terminated.

File Descriptors

Type

Severity

Activity Name

Attributes

Description

Event

Info

File Descriptor Open

Process, File descriptors

A new file descriptor was opened.

Event

Info

File Descriptor Close

Process, File descriptors

A file descriptor was closed.

Alert

Medium

File Descriptor File Content Change

Process, File descriptors

A file descriptor with write permissions was closed, and its content might have changed.

GPU Telemetry

GPU telemetry was added in DOCA Argus 3.5 and is opt-in. To enable it, run a command such as nvidia-smi rusd -spm all.

Type

Severity

Activity Name

Attributes

Description

Event

Info

GPU Identity

GPU identity

Identifies the make and model of each GPU on the AI node. By default, DOCA Argus generates one event per GPU every minute.

Event

Info

GPU Metrics

GPU metrics

Provides telemetry for each GPU in operation. By default, DOCA Argus generates one event per GPU every second.

Event

Info

GPU Topology

GPU topology

Provides information about how devices are connected to each other. By default, DOCA Argus generates one event every minute.

Kernel Modules

Kernel module auditing was added in DOCA Argus 3.5.

Type

Severity

Activity Name

Attributes

Description

Event

Info

Kernel Module Changed

Kernel modules

Provides information about a kernel module whose parameters changed.

Event

Info

Kernel Module Loaded

Kernel modules

On initialization, lists all loaded kernel modules. After that, provides information about each newly loaded kernel module.

Event

Info

Kernel Module Unloaded

Kernel modules

Provides information about a kernel module that was unloaded.

Network Connections

Type

Severity

Activity Name

Attributes

Description

Event

Info

Network Connection Created

Process, TCP network connections, File descriptors, Threads

A new TCP network connection was created.

Event

Info

Network Connection Terminated

Process, TCP network connections, File descriptors, Threads

A TCP network connection was terminated.

Alert

Low

TCP Connection Excessive Data

Process, TCP network connections, File descriptors, Threads

Monitors the amount of incoming and outgoing data, in bytes, on a TCP network connection, and reports when it exceeds a size threshold. You can configure the threshold separately for incoming and outgoing traffic.

Alert

Low

TCP Long Lasting Connection

Process, TCP network connections, File descriptors, Threads

Monitors the overall duration of a TCP connection, and reports when it exceeds a configurable time threshold.

Event

Info

TCP Network Connection State Change

Process, TCP network connections, File descriptors, Threads

Monitors for changes in the state of TCP network connections, for example, SYN SENT or SYN RECEIVED.

Event

Info

TCP Network Connections Status

Process, TCP network connections, File descriptors, Threads

Periodically reports the TCP connections currently open for each process, including, for example, the number of packets and bytes sent and received. The reporting interval is configurable. Disabled by default.

Alert

High

Reverse Shell Detected

Process, TCP network connections, File descriptors, Threads

Monitors for a process that started with stream redirection to a remote connected socket, where stdin is bound to a remote socket.

Process Memory

Type

Severity

Activity Name

Attributes

Description

Event

Info

Process Memory Created

Process, Process memory

A new virtual memory area, such as a heap, stack, or executable area, was created. Disabled by default.

Event

Info

Process Memory Terminated

Process, Process memory

A virtual memory area is no longer visible (terminated). Disabled by default.

Event

Warning

New Executable Anonymous Memory Mapped

Process, Process memory

An executable anonymous memory area was mapped.

Alert

Medium

Executable Permissions Added

Process, Process memory

Executable permissions were added to a memory area.

Alert

Medium

Executable Permissions Removed

Process, Process memory

Executable permissions were removed from a memory area.

Event

Info

New File Mapped

Process, Process memory, Runtime manifest

A new memory-mapped file was detected.

Event

Info

File Unmapped

Process, Process memory

A memory-mapped file was unmapped.

Processes

Type

Severity

Activity Name

Attributes

Description

Event

Info

Process Created

Process, including complete information about the parent process

A new process was detected.

Event

Info

Process Terminated

Process

A process was terminated.

Event

Warning

Process Zombie

Process (partial)

Detects a process in a zombie state.

Alert

High

Process Hidden

Process

Detects a process in a hidden state.

Runtime Manifest

Type

Severity

Activity Name

Attributes

Description

Alert

High

Foreign Binary Executed

Process, Process memory, Runtime manifest

Monitors for an executed binary that was not part of the original container image or was modified from it. This might indicate that an attacker controls the workload and is running arbitrary commands.

Alert

High

Binary Executed Not as Intended

Process, Process memory, Runtime manifest, Expected behavior

Monitors for an executed binary from the original container image that ran with command-line arguments or from a folder path that does not match the original container image.

Alert

High

Foreign Binary Executed File Size Mismatch

Process, Process memory, Runtime manifest, Expected behavior

Monitors for an executed binary whose reported file size differs from the file size of the binary in the original container image.

Alert

High

Foreign Library Loaded

Process, Process memory, Runtime manifest

Monitors for a library loaded into memory that was not part of the original container image or was modified from it. This might indicate that an attacker controls the workload and is running arbitrary code.

Alert

High

Foreign Library Loaded File Size Mismatch

Process, Process memory, Runtime manifest, Expected behavior

Monitors for a loaded library whose reported file size differs from the file size of the library in the original container image.

Threads

Type

Severity

Activity Name

Attributes

Description

Event

Info

Thread Created

Process, Threads

A new thread was created.

Event

Info

Thread Terminated

Process, Threads

A thread was terminated.

System Activity Messages

System activity messages report the operational state of the DOCA Argus service. Their message type is SYSTEM_ACTIVITY.

Severity

Activity Name

Attributes

Description

Error

Details Gathering Failed

Process (partial)

DOCA Argus failed to collect information.

Medium

File Descriptor Limit Reached

—

The limit on the number of file descriptors to operate on was reached.

Error

Host Initialization Failed

—

Workload detection failed.

Info

Host Initialization Started

—

Workload detection started.

Info

Host Initialization Successful

—

Workload detection completed successfully.

Error

Invalid SBOM Configuration File

—

The provided SBOM configuration file is invalid. Added in DOCA Argus 3.5.

Error

Invalid SBOM Configuration Folder

—

The configured folder for the SBOM configuration file is invalid. Added in DOCA Argus 3.5.

Medium

Kernel Module Limit Reached

—

The limit on the number of kernel modules to operate on was reached. Added in DOCA Argus 3.5.

Info

Load Memory Regions Failed

—

DOCA Argus failed to load the memory regions information. Added in DOCA Argus 3.5.

Info

Load Memory Regions Successful

—

DOCA Argus loaded the memory regions information. Added in DOCA Argus 3.5.

Info

Loading Profile Candidate

—

DOCA Argus identified an OS profile to use.

Error

No Matching Profile Found

—

No matching OS profile was found.

Info

OS Identifier Discovery Extended

—

Discovery of the underlying OS of the workload is taking longer than expected.

Info

OS Identifier Found

—

DOCA Argus detected the underlying OS of the workload.

Medium

Process Limit Reached

—

The limit on the number of processes to operate on was reached.

Medium

Process Memory Limit Reached

Process

The limit on the number of virtual address descriptors to operate on was reached.

Error

Profile Parsing Failed

—

DOCA Argus failed to parse an OS profile.

Info

Profile Verification Successful

—

DOCA Argus initialized successfully with a specific OS profile.

Error

Profile Verification Failed

—

Initialization with an OS profile failed. DOCA Argus continues and tries the next OS profile candidates.

Info

Service Initialization Started

—

The DOCA Argus initialization stage started.

Info

Service Initialization Successful

—

The DOCA Argus initialization stage completed successfully.

Error

Service Initialization Failed

—

DOCA Argus initialization failed.

Error

Service Runtime Failure

—

A critical internal service error occurred. DOCA Argus is offline.

Info

Service Gracefully Shutdown

—

DOCA Argus shut down gracefully after a user request.

Medium

Threads Limit Reached

Process

The limit on the number of threads to operate on was reached.

Error

Unable to Determine Target OS

—

DOCA Argus failed to detect the underlying OS of the workload.

Attributes

This section describes the attribute objects that DOCA Argus reports and their fields.

Bash Shell Command Attributes

Bash shell command execution data is reported in the shell_command_details object:

Attribute

Description

shell_command_execution_history

A list, in JSON format, of executed commands (command_line) and their execution times (command_execution_time_iso_8601_ns).

command_execution_time_iso_8601_ns

Command execution time.

command_line

The command line.

Container Attributes

Container information is reported in the container_details object:

Attribute

Description

container_id

The container's unique identifier.

container_name

The container name.

container_image_name

The container image name.

container_image_digest

The SHA-256 hash that uniquely identifies a specific build of a container image.

container_image_manifest_digest

A unique cryptographic identifier (SHA-256 hash) that represents the content of a Docker image.

pod_name

The name of the Pod that the container is part of.

pod_uid

The unique identifier of the Pod that the container is part of.

pod_ip_address

The IP address of the Pod that the container is part of.

enrichment_status

complete or partial. Indicates whether the information collection process completed.

File Descriptor Attributes

File descriptor information is reported in the file_descriptor_details object:

Attribute

Description

file_descriptor_index

The file descriptor's identifier.

file_descriptor_name

The file descriptor's name.

file_descriptor_file_type

The file descriptor's type, for example, socket, regular file, or pipe.

file_descriptor_file_mode

How the current process opened the file (read-only, write-only, or read-write) and which status flags were assigned to it (for example, append or non-block). For example: READ_WRITE|NON_BLOCKING.

file_descriptor_inode_number

The unique identifier of the inode associated with the file descriptor.

file_descriptor_inode_device_identifier

Identifies the device or file system on which the inode (the file) resides, as a major and minor number. The major number identifies the device type, and the minor number identifies a specific instance of that device.

file_descriptor_inode_permissions

File permissions.

file_descriptor_inode_uid

The user ID of the file's owner.

file_descriptor_inode_gid

The group ID of the file's owner.

file_descriptor_inode_size_bytes

The size of the file, in bytes.

file_descriptor_inode_deleted

Indicates whether the file was deleted.

file_descriptor_inode_hash_sha1

The SHA-1 hash of the file. Calculated only for File Descriptor File Content Change alerts.

file_descriptor_inode_hash_sha256

The SHA-256 hash of the file. Calculated only for File Descriptor File Content Change alerts.

file_descriptor_inode_hash_md5

The MD5 hash of the file. Calculated only for File Descriptor File Content Change alerts.

file_descriptor_thread_id_array

The identifiers of the threads that use the file descriptor.

file_descriptor_inode_change_time_iso_8601_ns

Timestamp of the last inode change.

file_descriptor_inode_modify_time_iso_8601_ns

Timestamp of the last modification of the file content.

file_descriptor_inode_access_time_iso_8601_ns

Timestamp of the last access to the file content.

GPU Telemetry Attributes

GPU Identity, GPU Metrics, and GPU Topology events include the attribute objects in this section.

GPU Identity

GPU Identity events include the attribute objects in this section.

GPU Details

The gpu_details object includes the following attributes:

Attribute

Description

gpu_uuid

Universally unique identifier (UUID) assigned to the GPU, providing a persistent identifier across system reboots.

gpu_uuid_source

Source from which the GPU UUID was obtained (for example, hardware, driver, or management interface).

gpu_pci_bus_id

PCI Bus ID of the GPU in the format domain:bus:device.function, uniquely identifying its location on the PCIe bus.

gpu_index

Zero-based index assigned to the GPU by the driver on the host system.

gpu_device_minor

Minor device number associated with the GPU device node in the operating system (for example, /dev/nvidiaX).

gpu_display_name

Human-readable model name of the GPU as reported by the driver.

gpu_product_brand

NVIDIA product brand associated with the GPU (for example, GeForce, RTX, Quadro, Tesla, NVIDIA RTX, Data Center).

gpu_product_architecture

NVIDIA GPU architecture generation (for example, Kepler, Maxwell, Pascal, Volta, Turing, Ampere, Ada, Hopper, Blackwell).

gpu_gsp_mode

Current status or operating mode of the GPU System Processor (GSP) firmware.

gpu_compute_mode

Compute mode that determines how compute contexts can access the GPU (for example, Default, Exclusive Process, Prohibited).

gpu_compute_capability

CUDA Compute Capability version supported by the GPU, indicating its programming and hardware feature set (for example, 8.0, 9.0).

gpu_sm_version

Streaming Multiprocessor (SM) architecture version implemented by the GPU.

gpu_pdi

Persistent Device Identifier (PDI), a stable identifier used to uniquely identify the physical GPU across software and management interfaces.

gpu_vbios_version

Version of the GPU Video BIOS (VBIOS) firmware installed on the device.

gpu_board_id

Identifier assigned to the physical GPU board, used to distinguish boards within a system.

gpu_board_id_source

Source from which the GPU board identifier was obtained (for example, hardware, firmware, or driver).

gpu_part_number

Manufacturer-assigned part number identifying the GPU board or module.

gpu_board_serial_number

Manufacturer-assigned serial number uniquely identifying the physical GPU board.

GPU Inforom Board Info Details

The gpu_inforom_board_info_details object includes the following attributes:

Attribute

Description

marketing_name

Commercial marketing name of the GPU product as assigned by the manufacturer (for example, NVIDIA H100 PCIe 80GB).

serial_number

Manufacturer-assigned serial number uniquely identifying the physical GPU board.

product_part_number

Manufacturer product part number (PPN) identifying the GPU board or module configuration.

board_699_part_number

NVIDIA internal 699-series board part number identifying the board assembly.

board_965_part_number

NVIDIA internal 965-series board part number identifying the board or manufacturing configuration.

board_revision

Hardware revision level of the GPU board.

memory_part_id

Manufacturer-assigned identifier for the memory devices installed on the GPU board.

memory_date_code

Manufacturing date code for the GPU memory devices.

build_date_iso_860

GPU board manufacturing date formatted as an ISO 8601 timestamp.

build_date_raw

Raw manufacturing date value reported by the hardware or firmware.

memory_manufacturer_raw

Raw identifier of the GPU memory manufacturer as reported by the hardware or firmware.

memory_manufacturer_label

Human-readable name of the GPU memory manufacturer derived from the raw identifier (for example, Samsung, SK hynix, Micron).

board_type_raw

Raw identifier indicating the GPU board type as reported by the hardware or firmware.

board_type_label

Human-readable description of the GPU board type derived from the raw identifier.

GPU Graphics Core Counts Details

The gpu_graphics_core_counts_details object is an array. Each entry includes the following attributes:

Attribute

Description

graphics_index

Zero-based index identifying the GPU graphics device within the system or graphics subsystem.

sm_version

Streaming Multiprocessor (SM) architecture version implemented by the GPU, indicating the generation of SM hardware features.

gpc_count

Number of Graphics Processing Clusters (GPCs) present on the GPU.

tpc_count

Total number of Texture Processing Clusters (TPCs) across all Graphics Processing Clusters (GPCs).

sm_count

Total number of Streaming Multiprocessors (SMs) available on the GPU.

sm_per_tpc

Number of Streaming Multiprocessors (SMs) contained within each Texture Processing Cluster (TPC).

gpu_core_count

Total number of programmable GPU processing cores (CUDA cores) available on the GPU.

gpu_cores_per_sm

Number of GPU processing cores (CUDA cores) implemented within each Streaming Multiprocessor (SM).

tensor_core_count

Total number of Tensor Cores available on the GPU for accelerating matrix operations used in AI and high-performance computing workloads.

rt_core_count

Total number of Ray Tracing (RT) Cores available on the GPU for hardware-accelerated ray tracing operations.

rop_unit_count

Total number of Render Output Pipeline (ROP) units responsible for pixel blending, raster operations, and framebuffer writes.

rop_operations_count

Total number of ROP operations that can be performed in parallel by the GPU, reflecting the aggregate pixel output capability of the ROP subsystem.

GPU Metrics

GPU Metrics events include the gpu_details object, with the same attributes as in GPU Details, and the attribute objects in this section.

GPU Utilization Details

The gpu_utilization_details object includes the following attributes:

Attribute

Description

perf_dev_util_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the device utilization metrics were collected.

perf_dev_util_timestamp_kind

Indicates the type or source of the timestamp (for example, device time, host time, synchronized clock).

perf_dev_util_stale

Boolean flag indicating whether the device utilization metrics are stale or no longer reflect the latest sampling period.

gpu_percent_busy

Percentage of time the GPU compute and graphics engines were actively processing work during the sampling interval.

memory_percent_busy

Percentage of time the GPU memory subsystem was actively servicing memory operations during the sampling interval.

nvenc_clk_percent_busy

Percentage utilization of the NVIDIA hardware video encoder (NVENC) engine during the sampling interval.

nvenc_sampling_period_us

Duration of the sampling interval for NVENC utilization metrics, in microseconds.

nvdec_clk_percent_busy

Percentage utilization of the NVIDIA hardware video decoder (NVDEC) engine during the sampling interval.

nvdec_sampling_period_us

Duration of the sampling interval for NVDEC utilization metrics, in microseconds.

nvjpg_clk_percent_busy

Percentage utilization of the NVIDIA JPEG (NVJPG) hardware acceleration engine during the sampling interval.

nvjpg_sampling_period_us

Duration of the sampling interval for NVJPG utilization metrics, in microseconds.

nvofa_clk_percent_busy

Percentage utilization of the NVIDIA Optical Flow Accelerator (NVOFA) during the sampling interval.

nvofa_sampling_period_us

Duration of the sampling interval for NVOFA utilization metrics, in microseconds.

proc_util_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the per-process utilization metrics were collected.

proc_util_timestamp_kind

Indicates the type or source of the per-process utilization timestamp.

proc_util_stale

Boolean flag indicating whether the per-process utilization metrics are stale.

proc_util_samples

An array of per-process GPU utilization samples captured during the reporting interval.

Each entry in the proc_util_samples array includes the following attributes:

Attribute

Description

sample_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the individual process utilization sample was recorded.

process_id

Operating system process identifier (PID) associated with the GPU workload.

sub_process_id

Identifier of a subprocess or GPU execution context associated with the parent process, if applicable.

graphics_util_percent

Percentage of GPU graphics engine utilization attributed to the process during the sampling interval.

framebuffer_util_percent

Percentage of GPU framebuffer (memory subsystem) utilization attributed to the process during the sampling interval.

nvenc_util_percent

Percentage of NVENC engine utilization attributed to the process during the sampling interval.

nvdec_util_percent

Percentage of NVDEC engine utilization attributed to the process during the sampling interval.

nvjpg_util_percent

Percentage of NVJPG engine utilization attributed to the process during the sampling interval.

nvofa_util_percent

Percentage of NVOFA engine utilization attributed to the process during the sampling interval.

GPU Clock Details

The gpu_clocks_details object includes the following attributes:

Attribute

Description

clk_public_domain_infos_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the GPU clock domain information was collected.

clk_public_domain_infos_timestamp_kind

Indicates the type or source of the clock domain information timestamp (for example, host time, device time).

clk_public_domain_infos_stale

Boolean flag indicating whether the clock domain information is stale or no longer reflects the current GPU state.

clk_public_domain_infos_supported_by_driver

Boolean indicating whether GPU clock domain information is supported by the installed driver.

clk_public_domain_infos_available

Boolean indicating whether clock domain information is currently available from the GPU.

clock_graphics_target_mhz

Target operating frequency of the GPU graphics clock domain, in megahertz (MHz).

clock_graphics_actual_khz

Actual operating frequency of the GPU graphics clock domain, in kilohertz (kHz).

clock_memory_target_mhz

Target operating frequency of the GPU memory clock domain, in megahertz (MHz).

clock_memory_actual_khz

Actual operating frequency of the GPU memory clock domain, in kilohertz (kHz).

clock_video_target_mhz

Target operating frequency of the GPU video engine clock domain, in megahertz (MHz).

clock_video_actual_khz

Actual operating frequency of the GPU video engine clock domain, in kilohertz (kHz).

clock_sm_target_mhz

Target operating frequency of the Streaming Multiprocessor (SM) clock domain, in megahertz (MHz).

clock_sm_actual_khz

Actual operating frequency of the Streaming Multiprocessor (SM) clock domain, in kilohertz (kHz).

perf_current_pstate_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the current GPU performance state (P-state) was reported.

perf_current_pstate_timestamp_kind

Indicates the type or source of the performance state timestamp.

perf_current_pstate_stale

Boolean flag indicating whether the reported performance state is stale.

perf_current_pstate_raw

Raw numeric identifier of the current GPU performance state (P-state) as reported by the driver or firmware.

perf_current_pstate_label

Human-readable label of the current GPU performance state (for example, P0, P2, P8), where lower-numbered states generally represent higher performance.

clk_throttle_info_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the GPU clock throttling information was collected.

clk_throttle_info_timestamp_kind

Indicates the type or source of the clock throttling information timestamp.

clk_throttle_info_stale

Boolean flag indicating whether the clock throttling information is stale.

clk_throttle_reason_mask_raw

Raw bitmask representing all GPU clock throttling reasons currently reported by the driver or firmware.

clk_throttle_reasons_active

List of active GPU clock throttling reasons currently limiting clock frequencies (for example, thermal, power, idle, application clocks, synchronization boost).

clk_throttle_reference_time_iso_8601_ns

Reference timestamp (ISO 8601, nanosecond precision) used as the baseline for reporting accumulated clock throttling statistics.

clk_throttle_supported_violation_time_mask

Bitmask indicating which clock throttling violation-time counters are supported by the GPU and driver.

GPU Thermal Details

The gpu_thermal_details object includes the following attributes:

Attribute

Description

temperature_entries

An array of temperature measurements reported by the GPU for one or more monitored temperature sensors.

fan_cooler_status_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the fan speed information was collected.

fan_cooler_status_timestamp_kind

Indicates the type or source of the fan status timestamp.

fan_cooler_status_stale

Boolean flag indicating whether the reported fan status is stale.

fan_cooler_status_rpms

Rotational speed of each GPU cooling fan, expressed in revolutions per minute (RPM). For GPUs with multiple fans, this field contains the RPM for each individual fan.

Each entry in the temperature_entries array includes the following attributes:

Attribute

Description

temperature_type

Type of temperature sensor associated with the measurement (for example, GPU core, memory, hotspot, board, VRM).

temperature_celsius

Temperature reported by the sensor, expressed in degrees Celsius (°C).

temperature_raw

Raw temperature value reported by the hardware or firmware before any formatting or unit conversion.

temperature_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the temperature measurement was collected.

temperature_timestamp_kind

Indicates the type or source of the temperature measurement timestamp (for example, host time, device time).

temperature_stale

Boolean flag indicating whether the temperature measurement is stale or no longer reflects the current sensor reading.

GPU Power Details

The gpu_power_details object includes the following attributes:

Attribute

Description

avg_power_usage_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the average power measurements were collected.

avg_power_usage_timestamp_kind

Indicates the type or source of the average power measurement timestamp (for example, host time, device time).

avg_power_usage_stale

Boolean flag indicating whether the average power measurements are stale.

power_avg_gpu_milliwatts

Average power consumed by the GPU silicon during the sampling interval, expressed in milliwatts (mW).

power_avg_module_milliwatts

Average total power consumed by the GPU module (including GPU and other module components), expressed in milliwatts (mW).

power_avg_memory_milliwatts

Average power consumed by the GPU memory subsystem during the sampling interval, expressed in milliwatts (mW).

inst_power_usage_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the instantaneous power measurements were collected.

inst_power_usage_timestamp_kind

Indicates the type or source of the instantaneous power measurement timestamp.

inst_power_usage_stale

Boolean flag indicating whether the instantaneous power measurements are stale.

power_inst_gpu_milliwatts

Instantaneous power consumption of the GPU silicon, expressed in milliwatts (mW).

power_inst_module_milliwatts

Instantaneous total power consumption of the GPU module, expressed in milliwatts (mW).

power_inst_cpu_milliwatts

Instantaneous power consumption of the associated CPU or CPU subsystem, expressed in milliwatts (mW), when supported by the platform.

power_limit_gpu_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the GPU power limit information was collected.

power_limit_gpu_timestamp_kind

Indicates the type or source of the GPU power limit timestamp.

power_limit_gpu_stale

Boolean flag indicating whether the GPU power limit information is stale.

power_limit_requested_milliwatts

Configured (requested) GPU power limit, expressed in milliwatts (mW).

power_limit_enforced_milliwatts

Effective GPU power limit currently enforced by the hardware or firmware, expressed in milliwatts (mW).

power_policy_status_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the GPU power policy status was collected.

power_policy_status_timestamp_kind

Indicates the type or source of the power policy status timestamp.

power_policy_status_stale

Boolean flag indicating whether the GPU power policy status is stale.

power_target_gpu_milliwatts

Target GPU power level established by the active power management policy, expressed in milliwatts (mW).

GPU Memory Details

The gpu_memory_details object includes the following attributes:

Attribute

Description

pma_memory_info_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the PMA memory information was collected.

pma_memory_info_timestamp_kind

Indicates the type or source of the PMA memory information timestamp (for example, host time, device time).

pma_memory_info_stale

Boolean flag indicating whether the PMA memory information is stale.

pma_memory_info_seq

Sequence number associated with the PMA memory information snapshot, used to identify or order successive updates.

pma_total_bytes

Total amount of GPU memory managed by the PMA (Physical Memory Allocator), expressed in bytes.

pma_free_bytes

Amount of PMA-managed GPU memory currently available for allocation, expressed in bytes.

bar1_memory_info_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the BAR1 memory information was collected.

bar1_memory_info_timestamp_kind

Indicates the type or source of the BAR1 memory information timestamp.

bar1_memory_info_stale

Boolean flag indicating whether the BAR1 memory information is stale.

bar1_memory_info_seq

Sequence number associated with the BAR1 memory information snapshot, used to identify or order successive updates.

bar1_size_bytes

Total size of the GPU BAR1 aperture, expressed in bytes. BAR1 provides CPU access to GPU memory through PCIe address mapping.

bar1_avail_size_bytes

Amount of BAR1 address space currently available for new memory mappings, expressed in bytes.

mem_row_remap_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the memory row remapping information was collected.

mem_row_remap_timestamp_kind

Indicates the type or source of the memory row remapping timestamp.

mem_row_remap_stale

Boolean flag indicating whether the memory row remapping information is stale.

mem_row_remap_histogram_max

Number of memory banks that have reached the maximum allowable number of remapped memory rows.

mem_row_remap_histogram_high

Number of memory banks with a high utilization of available row remapping capacity.

mem_row_remap_histogram_partial

Number of memory banks that have partially utilized their row remapping capacity.

mem_row_remap_histogram_low

Number of memory banks with low row remapping utilization.

mem_row_remap_histogram_none

Number of memory banks with no remapped memory rows.

mem_row_remap_correctable_rows

Total number of memory rows remapped due to correctable memory errors.

mem_row_remap_uncorrectable_rows

Total number of memory rows remapped due to uncorrectable memory errors.

mem_row_remap_correctable_rows_inactive

Number of remapped rows associated with correctable errors that are currently inactive or not in use.

mem_row_remap_uncorrectable_rows_inactive

Number of remapped rows associated with uncorrectable errors that are currently inactive or not in use.

mem_row_remap_is_pending

Boolean indicating whether one or more memory row remapping operations are pending and require a future reset or maintenance event to take effect.

mem_row_remap_has_failure_occurred

Boolean indicating whether the GPU has experienced a memory row remapping failure, indicating that one or more faulty memory rows could not be successfully remapped.

Use the following information to interpret these metrics:

  • PMA (Physical Memory Allocator) metrics describe the internal physical memory allocation state of the GPU.

  • BAR1 metrics describe the PCIe Base Address Register (BAR1) aperture that the CPU uses to map and directly access GPU memory.

  • Memory row remapping metrics indicate GPU memory health and reliability. NVIDIA GPUs transparently remap failing DRAM rows to spare rows to maintain correct operation. These counters help identify degrading memory devices before they become critical.

GPU ECC Details

The gpu_ecc_details object includes the following attributes:

Attribute

Description

mem_ecc_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the ECC memory statistics were collected.

mem_ecc_timestamp_kind

Indicates the type or source of the ECC statistics timestamp (for example, host time, device time).

mem_ecc_stale

Boolean flag indicating whether the ECC statistics are stale.

mem_ecc_total_corrected_volatile

Total number of corrected ECC errors recorded since the last driver initialization or system reboot (volatile count).

mem_ecc_total_corrected_aggregate

Lifetime cumulative number of corrected ECC errors retained across resets (aggregate count).

mem_ecc_total_uncorrected_volatile

Total number of uncorrected ECC errors recorded since the last driver initialization or system reboot (volatile count).

mem_ecc_total_uncorrected_aggregate

Lifetime cumulative number of uncorrected ECC errors retained across resets (aggregate count).

mem_ecc_dram_corrected_volatile

Volatile count of corrected ECC errors detected in GPU DRAM.

mem_ecc_dram_corrected_aggregate

Lifetime cumulative count of corrected ECC errors detected in GPU DRAM.

mem_ecc_dram_uncorrected_volatile

Volatile count of uncorrected ECC errors detected in GPU DRAM.

mem_ecc_dram_uncorrected_aggregate

Lifetime cumulative count of uncorrected ECC errors detected in GPU DRAM.

mem_ecc_sram_corrected_volatile

Volatile count of corrected ECC errors detected in GPU SRAM structures.

mem_ecc_sram_corrected_aggregate

Lifetime cumulative count of corrected ECC errors detected in GPU SRAM structures.

mem_ecc_sram_uncorrected_volatile

Volatile count of uncorrected ECC errors detected in GPU SRAM structures.

mem_ecc_sram_uncorrected_aggregate

Lifetime cumulative count of uncorrected ECC errors detected in GPU SRAM structures.

mem_ecc_volatile_sram_cor_uni

Number of unique corrected SRAM ECC error locations detected since the last reset (volatile).

mem_ecc_volatile_sram_unc_parity_uni

Number of unique uncorrected SRAM parity error locations detected since the last reset (volatile).

mem_ecc_volatile_sram_unc_secded_uni

Number of unique uncorrected SRAM SECDED (Single Error Correction, Double Error Detection) error locations detected since the last reset (volatile).

mem_ecc_volatile_dram_cor_tot

Total number of corrected DRAM ECC errors detected since the last reset (volatile).

mem_ecc_volatile_dram_unc_tot

Total number of uncorrected DRAM ECC errors detected since the last reset (volatile).

mem_ecc_client_sram_parity_uncorrected_unique

Number of unique uncorrected SRAM parity errors reported by the client ECC interface.

mem_ecc_client_sram_secded_uncorrected_unique

Number of unique uncorrected SRAM SECDED errors reported by the client ECC interface.

mem_ecc_client_sram_corrected_unique

Number of unique corrected SRAM ECC error locations reported by the client ECC interface.

mem_ecc_client_dram_uncorrected_total

Total number of uncorrected DRAM ECC errors reported by the client ECC interface.

mem_ecc_client_dram_corrected_total

Total number of corrected DRAM ECC errors reported by the client ECC interface.

mem_ecc_client_last_cleared_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the client ECC counters were last cleared.

mem_ecc_client_sram_error_threshold_exceeded

Boolean indicating whether the configured threshold for SRAM ECC errors has been exceeded.

mem_ecc_client_sram_bucket_l2

Number of SRAM ECC errors attributed to the GPU L2 cache subsystem.

mem_ecc_client_sram_bucket_sm

Number of SRAM ECC errors attributed to Streaming Multiprocessor (SM) SRAM structures.

mem_ecc_client_sram_bucket_pcie

Number of SRAM ECC errors attributed to the PCIe interface or associated SRAM resources.

mem_ecc_client_sram_bucket_firmware

Number of SRAM ECC errors attributed to firmware-managed SRAM resources.

mem_ecc_client_sram_bucket_other

Number of SRAM ECC errors attributed to SRAM resources not covered by the predefined categories.

shadow_err_cont_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the shadow error containment status was collected.

shadow_err_cont_timestamp_kind

Indicates the type or source of the shadow error containment timestamp.

shadow_err_cont_stale

Boolean flag indicating whether the shadow error containment information is stale.

shadow_err_cont_seq

Sequence number associated with the shadow error containment status snapshot.

shadow_err_cont_val

Current shadow error containment status or value reported by the GPU, indicating whether error containment mechanisms have been triggered or are active.

Use the following information to interpret these metrics:

  • Volatile counters reset when the GPU driver reloads or the system reboots.

  • Aggregate counters are lifetime cumulative error counts that the GPU maintains.

  • The hardware automatically repairs corrected ECC errors, which generally do not interrupt execution.

  • Uncorrected ECC errors cannot be repaired automatically and might cause application termination, page retirement, or GPU recovery actions.

  • SECDED (Single Error Correction, Double Error Detection) is the ECC algorithm commonly used in NVIDIA GPU memories.

  • Shadow error containment metrics report the status of hardware mechanisms that isolate or contain errors to prevent them from propagating through the GPU.

GPU PCIe Details

The gpu_pcie_details object includes the following attributes:

Attribute

Description

pci_bus_data_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the PCIe bus information was collected.

pci_bus_data_timestamp_kind

Indicates the type or source of the PCIe bus information timestamp (for example, host time, device time).

pci_bus_data_stale

Boolean flag indicating whether the PCIe bus information is stale or no longer reflects the current hardware state.

pcie_current_generation

Current negotiated PCI Express (PCIe) link generation (for example, Gen3, Gen4, Gen5) at which the GPU is operating.

pcie_link_cap_generation

Maximum PCIe generation supported by the current PCIe link, as determined by the platform and link capabilities.

pcie_gpu_cap_generation

Maximum PCIe generation supported by the GPU hardware.

pcie_speed_changes

Number of PCIe link speed changes or retraining events that have occurred since system initialization or the last counter reset.

pcie_max_speed

Maximum supported PCIe link data rate for the current connection, expressed as a PCIe generation or transfer rate (GT/s).

pcie_max_width

Maximum negotiated PCIe link width (number of active PCIe lanes), such as x8 or x16.

pcie_aspm

Current PCIe Active State Power Management (ASPM) configuration or status (for example, Disabled, L0s, L1, L1.1, L1.2).

pcie_raw_register

Raw value of the PCIe configuration or capability register containing low-level PCIe link configuration information.

GPU Graphics Details

The gpu_graphics_details object includes the following attributes:

Attribute

Description

gr_info_timestamp_iso_8601_ns

ISO 8601 timestamp (nanosecond precision) indicating when the GPU graphics (GR) information was collected.

gr_info_timestamp_kind

Indicates the type or source of the graphics information timestamp (for example, host time, device time).

gr_info_stale

Boolean flag indicating whether the graphics information is stale or no longer reflects the current GPU state.

gr_info_b_ctxsw_logging_enabled

Boolean indicating whether GPU graphics context-switch (CTXSW) logging is enabled. When enabled, the GPU records context-switch events for debugging, profiling, or diagnostic purposes.

GPU Topology

GPU Topology events include the attribute objects in this section.

GPU Topology Details

The gpu_topology_details object includes the following attributes:

Attribute

Description

gpu_nvlink_available

Boolean indicating whether the GPU supports and has one or more NVIDIA NVLink connections available for high-speed interconnect with other GPUs or NVLink-enabled devices.

gpu_total_count

Total number of GPUs detected in the system.

nic_total_count

Total number of network interface controllers (NICs) detected in the system that are available for topology or affinity reporting.

nvme_total_count

Total number of NVMe storage devices detected in the system.

nic_truncated_count

Number of NIC devices omitted from the reported results because the maximum supported reporting limit was exceeded.

nvme_truncated_count

Number of NVMe storage devices omitted from the reported results because the maximum supported reporting limit was exceeded.

GPU Topology Device Entries

The gpu_topology_device_entries object is an array. Each device type includes a different set of attributes, as listed in the Device Type column:

Attribute

Device Type

Description

device_label

All

Human-readable label assigned to the device for identification and inventory purposes.

device_kind

All

Type or category of the device (for example, GPU, NIC, NVMe storage, DPU).

device_pci_bus_id

All

PCI Bus ID of the device in the format domain:bus:device.function, uniquely identifying its location on the PCIe bus.

gpu_uuid

GPU

Universally unique identifier (UUID) assigned to the GPU, providing a persistent identifier across system reboots and driver reloads.

nic_ibdev_name

NIC

InfiniBand device name assigned by the operating system (for example, mlx5_0), identifying the RDMA-capable network interface.

nic_aliases

NIC

Collection of alternate names or aliases associated with the network interface, including operating system or management identifiers.

nic_netdev_name

NIC

Operating system network interface name (for example, eth0, ens1f0, ib0) used for network configuration and traffic management.

nic_pci_slot

NIC

PCI slot designation of the network interface card (NIC), identifying its physical or logical installation location in the system.

nvme_namespace_name

NVMe

Operating system name of the NVMe namespace or block device (for example, /dev/nvme0n1) representing a logical storage namespace on an NVMe drive.

GPU Topology Matrix Entries

The gpu_topology_matrix_entries object is an array. Each entry describes the connectivity between two GPUs, or between a GPU and another system device, and includes the following attributes:

Attribute

Description

row_label

Label identifying the source GPU or device represented by the row in a GPU topology or connectivity matrix.

column_label

Label identifying the destination GPU or device represented by the column in a GPU topology or connectivity matrix.

path_classification

Classification of the communication path between the devices identified by the row and column labels (for example, NVLink, NVSwitch, PCIe, same board, same NUMA node, system interconnect), indicating the type and relative performance of the connection.

GPU Topology Affinity Entries

The gpu_topology_affinity_entries object is an array. Each entry includes the following attributes:

Attribute

Description

device_label

Human-readable label assigned to the GPU device for identification in the system, for example, GPU0 or GPU1.

cpu_affinity

CPU cores or CPU affinity mask that are topologically closest to the GPU and recommended for scheduling GPU-related workloads. Identifies the processor cores with the best communication path to the GPU, which is useful for optimizing application thread placement.

numa_affinity

NUMA nodes to which the GPU is most closely connected, indicating the preferred locality for CPU memory allocations and execution.

closest_cpu_numa

Identifier of the CPU NUMA node with the lowest-latency connection to the GPU.

gpu_numa_id

NUMA node identifier assigned to the GPU when the GPU exposes memory as a NUMA node (for example, on systems that support GPU NUMA or HMM). Platform-dependent; typically available on systems that expose GPU memory through NUMA-aware operating system mechanisms.

pci_numa_locality

NUMA node associated with the GPU's PCIe root complex or PCIe attachment point, indicating the GPU's PCIe locality within the system topology.

Kernel Module Attributes

Kernel module information is reported in the kernel_module_details object:

Attribute

Description

kernel_module_name

Name of the loaded kernel module as registered with the Linux kernel.

kernel_module_size

Size of the loaded kernel module in memory, expressed in bytes.

kernel_module_state

Current operational state of the kernel module (for example, loaded, live, unloading).

kernel_module_address

Kernel virtual memory address at which the module is loaded.

kernel_module_parameters

Configurable parameters exposed by the kernel module, with their current values. Generally corresponds to the configurable options under /sys/module/<module_name>/parameters/.

kernel_module_used_by

Kernel modules or kernel subsystems that depend on or currently reference this module, including the module reference count when available. Similar to the dependency information that tools such as lsmod show.

kernel_module_taints

Kernel taint flags associated with the module, indicating conditions such as proprietary licensing, forced loading, unsupported configuration, or detected runtime issues. Corresponds to the Linux kernel taint mechanism (/proc/sys/kernel/tainted and module-specific taint flags), which indicates whether the kernel or module runs under conditions that might affect supportability or debugging.

kernel_module_source

Origin of the kernel module, such as the file system path, package, or provider from which the module was loaded.

TCP Network Connection Attributes

TCP network connection information is reported in the network_connection_details object:

Attribute

Description

file_descriptor_id

The unique identifier of the file descriptor associated with the socket.

connection_state

The TCP connection state.

protocol

The network protocol.

address_family

Defines the protocol suite and address structure that a socket or network interface uses to communicate. Tells the Linux kernel how to interpret IP addresses, manage routing tables, and format network packets at Layer 3 (network layer) and below. Added in DOCA Argus 3.5.2.

local_address

The IP address that, together with local_port, is bound to the host for a specific network endpoint, socket, or interface. Renamed in DOCA Argus 3.5.2.

local_port

The port that, together with local_address, is bound to the host for a specific network endpoint, socket, or interface. Renamed in DOCA Argus 3.5.2.

peer_address

The IP address that, together with peer_port, identifies the remote endpoint of a network connection or tunnel. Renamed in DOCA Argus 3.5.2.

peer_port

The port that, together with peer_address, identifies the remote endpoint of a network connection or tunnel. Renamed in DOCA Argus 3.5.2.

tcp_bytes_in

The amount of data received, in bytes.

tcp_bytes_out

The amount of data sent, in bytes.

tcp_segments_in

The number of TCP segments received.

tcp_segments_out

The number of TCP segments sent.

workload_network_interface_name

The interface name.

workload_network_interface_mac_address

The interface MAC address.

workload_network_interface_ipv4_address

The IPv4 addresses associated with the interface.

workload_network_interface_ipv6_address

The IPv6 addresses associated with the interface.

tcp_connection_creation_time_iso_8601_ns

The time when the TCP connection was observed, in ISO 8601 format with nanosecond precision.

tcp_connection_overall_duration_ns

The calculated overall duration of the TCP connection, in nanoseconds.

tcp_average_bytes_in

The average size of the received packets, in bytes.

tcp_average_bytes_out

The average size of the sent packets, in bytes.

Process Attributes

Process information is reported in the process_details object:

Attribute

Description

process_id

The process's unique identifier.

process_executable_path

The full path of the executed process, including the folder and the process filename.

process_name

The name of the executed binary.

process_self_exec_id

Thread-group change indicator. For example, this value increases on exec calls.

process_parent_process_id

The parent process's unique identifier.

process_cpu_clock_cycles

The number of CPU cycles consumed by the process.

process_real_group_id

The real group identifier of the process owner.

process_real_user_id

The real user identifier of the process owner.

process_effective_group_id

The effective group identifier of the process owner.

process_effective_user_id

The effective user identifier of the process owner.

process_command_line_arguments

The command-line arguments used to launch the process.

process_creation_time_iso_8601_ns

The process creation time, in ISO 8601 format with nanosecond precision.

process_state

The current state of the process.

process_pid_namespace

The namespace for the process identifiers.

process_mount_points_namespace

The namespace for the mount points.

process_network_namespace

The namespace for the network resources.

process_hash_sha256

The SHA-256 hash of the process's executed binary.

process_hash_sha1

The SHA-1 hash of the process's executed binary.

process_hash_md5

The MD5 hash of the process's executed binary.

process_file_size_bytes

The size of the process's executed binary, in bytes.

process_current_working_directory

The folder from which the process was executed.

process_container_id

The container identifier, if the process is part of a container.

Process Memory Attributes

Process memory information is reported in the process_memory_details object:

Attribute

Description

process_id

The associated process's unique identifier.

virtual_memory_area_start_address

The start address of the virtual memory area.

virtual_memory_area_end_address

The end address of the virtual memory area.

memory_permissions

The permissions associated with the virtual memory area.

is_main_process_executable

Indicates whether the virtual memory area belongs to the executed binary of the current process (the main executable).

file_path

The path, including the filename, of the file associated with the virtual memory area.

file_name

The filename associated with the virtual memory area.

Runtime Manifest Attributes

Runtime manifest, software bill of materials (SBOM), and process attestation information is reported in the process_attestation_details object:

Attribute

Description

elf_file_inode_number

The inode number of the ELF file.

elf_file_name

The name of the ELF file.

elf_file_folder_path

The folder path where the ELF file is located. Changed in DOCA Argus 3.5.

elf_file_hash_sha256

The SHA-256 hash value of the ELF file.

elf_file_hash_sha1

The SHA-1 hash value of the ELF file.

elf_file_size_bytes

The size of the ELF file, in bytes.

is_main_process_executable

Indicates whether this file is the executed binary of the current process (the main executable).

elf_file_type

The ELF file type.

elf_file_hash_md5

The MD5 hash value of the ELF file.

Expected Behavior Attributes

The Binary Executed Not as Intended, Foreign Binary Executed File Size Mismatch, and Foreign Library Loaded File Size Mismatch alerts also include the expected_behavior object:

Attribute

Description

process_hash_sha256

The SHA-256 hash of the executed binary or loaded library.

process_command_line_arguments

The command-line arguments used to launch the process.

process_folder_path

The folder that contains the executed process or library.

process_name

The name of the process or library.

process_file_size_bytes

The size of the executable binary or library, in bytes.

Thread Attributes

Thread information is reported in the thread_details object:

Attribute

Description

thread_id

The unique thread identifier.

thread_self_exec_id

Thread-group change indicator. For example, this value increases on exec calls.

thread_exit_state

The thread's exit state.

Last updated: