Networking UFM Telemetry Documentation

Prometheus Endpoint Support

Prometheus Endpoint

UFM Telemetry can expose a http or https endpoint to allow simple and effective integration with monitoring systems that work in poll mode and support Prometheus, CSV, or JSON data formats. The endpoint provides only the last data sample. The user cannot obtain statistics for time points in the past.

Supported Formats

A http endpoint provides data in Prometheus format by default. It also supports JSON and CSV formats. The user can request the desired format using a URL prefix, as shown in the table below.

Data Format

URL Prefix

Prometheus

-

JSON

/json

CSV

/csv

Data Filtering

A http endpoint can provide all sampled data using the default /metrics URL. The filtering functionality described in the Cset/Fset Filtering section is also supported. To use it place <name>.cset or <name>.fset file in appropriate folders. This folder should be stated in configuration file. See section Configuring Data Polling Endpoint for more details.

The Extended counter set filtering, as described below, presents an alternative approach to filtering functionality by enabling counters and field selection.

A filter file name is included in the URL to request that the data be filtered through the particular .cset/.fset/.xcset file the user intends. For example, if there are two filter files named name1.cset and name2.cset, then URLs /name1 (or /cset/name1 ) and /name2 (or /cset/name2) can be used to get filtered output described in these files accordingly.

The URL prefixes /cset, /fset and /xcset can also be used to specify which filter file is meant.

URL

File Extension

Folder Parameter in Configuration File

Note

/cset

*.cset

plugin_env_PROMETHEUS_CSET_DIR

If the cset folder is not explicitly specified in the configuration file, then the cset directory is set the same as the fset directory.

/fset

*.fset

plugin_env_PROMETHEUS_FSET_DIR

If the fset folder is not explicitly specified in the configuration file, then the fset directory is set the same as the cset directory.

/xcset

*.xcset

plugin_env_PROMETHEUS_XCSET_DIR

If the xcset folder is not explicitly specified in the configuration file, then the xcset directory is set the same as the fset directory.

If a URL prefix is not specified, then the filter file will be searched under both cset and fset folders. If they both have files with the same names, then both filters will be applied.

Extended Counter Set Filtering

The http server provides an optional Extended counter set (xcset) selection mechanism in addition to the counter set (cset) and field set (fset) filtering. The Extended Counterset allows the user to generate an output record which contains data from both ‘counters‘ and ‘event’ data records with the same index, which in the context of UFM Telemetry is generally the guid/port_num. To define an extended counter set, a file or group of files with the .xcset extension must be placed in its designated directory or adjacent to existing field or counter sets.

Each line of the file may contain:

  • Selection of a counter with an optional alias in the format “counter[=alias]”

  • Selection of a type’s field with an optional alias in the format “type.field[=alias]”

  • Reference to another file to be included “file.xcset”

Extended counter set files are searched for in the same directory as the source xcset.

Aliases are not mandatory, but if provided, they are used to name the selected counter or field in the output. Empty lines and comments that begin with the "#" sign are disregarded.

Rendering Hints

Extended counter sets support rendering hints to adjust attribution and representation of the metrics values.

The hints are given as a comma-separated list of key=value pairs following the field selection line after the semicolon ; character.

counter[=alias];key[=value][,key[=value]]*

For Example:

port_guid;label,hex,default=undefined
hw_port_state;lookup=printable_port_states

Supported rendering hints are the following:



Description


hex

n/a

Requests a numeric value to be rendered hexadecimal

port_num;hex

label

n/a

Attributes the field as Prometheus label

host_name;label

default

value

Sets a string value to be rendered in case of data for the field is missing

temperature;default=unknown

const

value

Add the marked field as constant value to the output

context;const=oberon

lookup

name

Use the named lookup table to replace the value when rendering.

hw_port_state;lookup=printable_port_states

URL Prefixes Priority

URL prefixes can be used to manipulate data output. It is important to use the prefixes in the correct order as they have assigned priorities. The table below shows URL prefixes priority assignments with examples:

Priority

Prefix

Link Examples

Description

1

/labels

/labels/metrics,

/metrics

Used to show labels from metadata files

2

/json, /csv

/json/metrics,

/csv/metrics,

/labels/json/metrics,

/labels/csv/metrics

Used to specify output format

3

/cset, /fset, /xcset

/cset/filter1,

/fset/filter2,

/labels/cset/filter1,

/labels/fset/filter2,

/json/cset/filter1,

/json/fset/filter2,

/csv/cset/filter1,

/csv/fset/filter2,

/csv/xcset/ib,

/labels/json/cset/filter1,

/labels/json/fset/filter2,

/labels/csv/cset/filter1,

/labels/csv/fset/filter2

Used to specify which type of filter file should be applied

Sharding Data Requests

The shard data filter is useful when metric scraping loads need to be distributed over time or across consumer space. Below are several examples of how to use sharding in queries.

  1. Shard counters and events by node_guid, serialize to CSV:

    curl -v  http://0.0.0.0:9352/csv/metrics?num_shards=2&shard=0&sharding_field=node_guid
    
  2. Shard by node_guid and filter by port number:

    curl -v  http://0.0.0.0:9352/csv/metrics?num_shards=2&shard=0&sharding_field=node_guid&port_num__eq__1
    
  3. Select a specific counter set explicitly:

    curl -v  http://0.0.0.0:9352/csv/cset/minimal?num_shards=2&shard=0&sharding_field=node_guid
    
  4. Use a predefined fieldset and shard by port number (using the port field):

    curl -v  http://0.0.0.0:9352/csv/fset/low_freq?num_shards=2&shard=0&sharding_field=port
    


Iteration-End Notification

The telemetry process can notify external HTTP endpoints when a data collection iteration ends.

A consumer can use this to know when fresh data is ready.

Configurations

Set the environment variables in launch_ibdiagnet_config.ini with the plugin_env_ prefix.

The notification is triggered whenUFM_TELEMETRY_NOTIFY_ENDPOINTS is set.

Variable

Default

Description

UFM_TELEMETRY_NOTIFY_ENDPOINTS

not set (disabled)

Comma-separated list of URLs to notify.

UFM_TELEMETRY_INSTANCE_ID

not set

Optional. If set, it is included as id in the notification payload.

URL format

  • Each URL must be http:// or https://, followed by a host, an optional port and an optional path. Example: http://localhost:8080/notify.

  • If no valid URL remains, the mechanism is disabled.

Notification Payload

Each notification is a HTTP POST to the configured URL with Content-Type: application/json.

Fields:

Field

Type

Present

Description

iter_status

boolean

always

true if the iteration finished successfully, false if it failed.

id

string

only if UFM_TELEMETRY_INSTANCE_ID is set

Telemetry instance identifier.

timestamp

integer

always

Time the notification was created, in microseconds since the Unix epoch.

Example

Successful iteration, instance ID not set:

{"iter_status":true,"timestamp": 1761483000123456}

Successful iteration, UFM_TELEMETRY_INSTANCE_ID=ufm-telemetry-1:

{"iter_status":true,"id":"ufm-telemetry-1","timestamp": 1761483000123456}


Direct CSV

Direct CSV is a lightweight way to export telemetry as CSV.

It streams data straight to the client instead of storing it in an additional in-memory cache first, so it much more memory efficient.

It supports CSV output only.

When to Use It

  • You only need CSV reports

Configuration

  • UFM Enterprise environment

    • Enable via gv.cfg

      • primary_direct_csv_url = no/yes (default = no)

      • secondary_direct_csv_url = no/yes (default = no)

  • UFM-T standalone
    plugin_env_CLX_PROMETHEUS_CACHE_ENABLED=false

  • Use 'Iteration-End Notification'

Access Example

curl 0.0.0.0:9002/direct/csv/xcset/low_freq_debug?scraper_id=1234

Part

Meaning

/direct/csv

Required prefix for all direct CSV requests

scraper_id

Identifies the client making the request

Notes

  • Every request must start with the /direct prefix.

  • Use a stable, unique scraper_id for each client. Data is delivered once per scraper per update. If the same scraper_id requests again before the data is refreshed, the response is empty. Wait for the next update interval before polling again.

  • Two clients sharing one scraper_id will cause one of them to get empty responses.

Configurable Indication of Missing Data

By default, numeric fields with no data are set to 0. This behavior can be modified to show 'N/A' instead.

Flag

Default Value

Effect

NA_INDICATION

false

Sets missing field values to 'N/A' instead of 0

NA_INDICATION?plugin_env_CLX_CLEAN_DATA_EVERY_ITER

false

Resets field values on each data collection iteration

Configuring Data Polling Endpoint

To configure the Prometheus endpoint, the keys listed below need to be set in the launch_ibdiagnet_config.ini file.

plugin_env_PROMETHEUS_ENDPOINT          	http://0.0.0.0:9100
plugin_env_PROMETHEUS_PROXY_ENDPOINT_PORT   9200
plugin_env_PROMETHEUS_INDEXES           	port_num
plugin_env_PROMETHEUS_FSET_INDEXES      	port,lid,guid,[CableInfo]^port_guid,^Port$
plugin_env_PROMETHEUS_CSET_DIR          	/config/prometheus_configs/cset
…

There are several options related to configuring the HTTP polling endpoint. The key plugin_env_PROMETHEUS_ENDPOINT is used to configure the IP interface for endpoint binding. The “0.0.0.0” part in the setting above means that any of the host's valid IP addresses can be used. Note that the user can also specify the host's IP address explicitly.

The plugin_env_PROMETHEUS_ENDPOINT key also configures the data transport. For regular HTTP, prefix to http. To send over a TLS connection, set the prefix to https, set the above mandatory parameters (keys), and select the existing security keys as follows.

A DH (key exchange protocol) file can also be specified if needed as follows:

plugin_env_CLX_SSL_DH_FILE=/certs/dh.pem

To use custom labels for Prometheus statistics, a metadata file is used. For details about labels and label file format, see sections Prometheus Labels and Prometheus Label Generation.

There are several options that allow configuring metadata. The file containing the labels used in Prometheus generation is set as follows:

plugin_env_CLX_METADATA_FILE=/config/labels.txt

The user can create the metadata file upon system setup or use a script to generate it automatically via script, using the following parameter:

plugin_env_CLX_METADATA_COMMAND=/opt/mellanox/collectx/telem/bin/gen_metadata --fabric compute --file /var/log/ibdiagnet2.ibnetdiscover --output /config/labels.txt

In the above example, the script generates metadata from /var/log/ibdiagnet2.ibnetdiscover. If the user wishes to create the label file manually, the above option should be commented out to prevent periodic overwriting of the content of the metadata file.

By default, the Prometheus endpoint provides statistics with the collection timestamps. The user can decide whether counter values will be passed with or without timestamps by setting the plugin_env_PROMETHEUS_SHOW_TIMESTAMPS parameter to T (true) or F (false), respectively. For example, to send counter values without timestamps, set the parameter as follows:

plugin_env_PROMETHEUS_SHOW_TIMESTAMPS=F

To use data filters folders with counter set, field sets, and extended counter sets, the directories where the files are stored should be configured as follows:

plugin_env_PROMETHEUS_CSET_DIR=/telemetry.config/prometheus_configs/cset
plugin_env_PROMETHEUS_FSET_DIR=/telemetry.config/prometheus_configs/fset
plugin_env_PROMETHEUS_XCSET_DIR=/telemetry.config/prometheus_configs/xcset

Any parameters not explicitly documented should not be changed and should be considered read-only.

Prometheus Labels

For use cases such as UFM Enterprise or UFM Cyber AI where the network topology is known, a human-readable name can be presented based on the GUID.

# TYPE PortXmitDataExtended counter
# TYPE PortXmitPktsExtended counter
PortXmitDataExtended{source="0x0002c90300f172a0", node_guid="2c90300f172a0", port_guid="2c90300f172a2", port_num="2"}  85554128244 1628683905941
PortXmitPktsExtended{source="0x0002c90300f172a0", node_guid="2c90300f172a0", port_guid="2c90300f172a2", port_num="2"}  1188251785  1628683905941

For integration with third-party applications, labels which are more human-readable may be generated using a labels metadata file, as described below.

Prometheus Label Generation

To generate custom labels, a file containing key-value pairs is used. When the keys are matched, the key-value pairs added to the Prometheus labels are generated.

The following is an example of the format of a labels metadata file:

ec0d9a0300b41a50_36|port_id|ec0d9a0300b41a50_36|device_name|SwitchIB Mellanox Technologies|device_type|switch|fabric|compute|hostname||node_desc||level|leaf|peer_level|server
ec0d9a0300b41a50_37|port_id|ec0d9a0300b41a50_37|device_name|SwitchIB Mellanox Technologies|device_type|switch|fabric|compute|hostname||node_desc||level|leaf|peer_level|
ec0d9a0300b41a58_1|port_id|ec0d9a0300b41a58_1|device_name||device_type|switch|fabric|compute|hostname|aggregation|node_desc|aggregation node|level||peer_level|leaf
98039b0300640b92_1|port_id|98039b0300640b92_1|device_name||device_type|host|fabric|compute|hostname|agx-1|node_desc|agx-1 mlx5_0|level|server|peer_level|leaf
98039b0300640c22_1|port_id|98039b0300640c22_1|device_name||device_type|host|fabric|compute|hostname|agx-2|node_desc|agx-2 mlx5_0|level|server|peer_level|leaf
0002c90300f172a0_2|port_id|0002c90300f172a0_2|device_name||device_type|host|fabric|compute|hostname|agx-3|node_desc|agx-3 mlx4_0|level|server|peer_level|leaf
98039b0300640b9a_1|port_id|98039b0300640b9a_1|device_name||device_type|host|fabric|compute|hostname|agx-3|node_desc|agx-3 mlx5_0|level|server|peer_level|leaf

The following is an example of the generated Prometheus output:

# TYPE infiniband_port_xmit_data_bytes counter
# TYPE infiniband_port_rcv_data_bytes counter
# TYPE infiniband_link_error_recovery_events counter
# TYPE infiniband_link_downed_events counter
# TYPE infiniband_cbw gauge
infiniband_port_xmit_data_bytes       {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}  82218360540 1628602711924
infiniband_port_rcv_data_bytes        {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}  82218429458 1628602711924
infiniband_link_error_recovery_events {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}            0 1628602711924
infiniband_link_downed_events         {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}            0 1628602711924
infiniband_cbw                        {port_id="0002c90300f172a0_2", ADDITIONAL_LABELS}}           0 1628602711924

where ADDITIONAL_LABELS include:
	hostname="agx-3"
	node_desc="agx-3 mlx5_0"
	device_name=""
	device_type="host"
	fabric="compute"
	level="server"
	peer_level="leaf"

To enable this functionality, the following additional keys need to be configured:

plugin_env_CLX_EXPORT_API_IBNETDISCOVER_RUN_ONCE 1    # Without this, the gen_metadata.py script cannot generate the human readable names, nor the level and peer_level.
plugin_env_CLX_METADATA_FILE /path/to/labels/file
plugin_env_CLX_METADATA_COMMAND "python3 /opt/mellanox/collectx/telem/bin/gen_metadata.py  --fabric compute --file /var/log/ibdiagnet2.ibnetdiscover -o /path/to/labels/file"

To test, the curl command can be used as follows:

[root@jazz11 /]# curl --silent  IP_ADDR_OF_HOST:9100/metrics |egrep   "xmit|rcv" |  tail
port_xmit_discard{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_rcv_switch_relay_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_rcv_constraint_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043
port_xmit_constraint_errors{device_name="",device_type="host",fabric="compute",hostname="jazz32",level="server",node_desc="jazz32 mlx5_2",peer_level="leaf",port_id="ec0d9a0300c04a54_1"}  0 1629194120043

Last updated: