Networking NVIDIA NVOS User Manual for InfiniBand Switches
Networking NVIDIA NVOS User Manual for InfiniBand Switches
Breadcrumbs

HCA Peer Port Telemetry

NVOS enables users to monitor telemetry for Host Channel Adapter (HCA) ports that are directly connected to the switch, without requiring access to the HCA itself.

Telemetry is streamed only for HCA ports with an active link. The available metrics align with the counters exposed for Infiniband interfaces and include:

  • Traffic statistics

  • Link status

  • Infiniband data

  • PHY diagnostics

How It Works

The HCA Peer Port Telemetry feature state can be set over NVOS using simple NVUE CLI commands:

Show command:

nvos@switch:~$ nv show system peer-port
          operational  applied
--------  -----------  -------
state     disabled     disabled
interval  5            5

Set command:

nvos@switch:~$ nv set system peer-port state <enabled | disabled>

Unset command:

nvos@switch:~$ nv unset system peer-port state

The state is disabled by default and the unset command will restore the state to disabled, if it is not already. 

  • By default, data is collected every 5 seconds.

  • Each discovered HCA port with an active link appears as a peer-port (hcaA2p1, hcaA2p2, and so on).

  • When an HCA link goes down, its peer-port entry is automatically removed from the output.

Telemetry data can be accessed using:

  • The nv show peer-port CLI command hierarchy

  • gNMI

Viewing HCA Peer-Ports

To view all active HCA peer-ports and their topology information, run the following command:

admin@nvos:~$ nv show peer-port
Peer Port   Port GUID           Node GUID           HCA Node Description           HCA Port Index  Switch Port
----------  ------------------  ------------------  -----------------------------  --------------  -----------
hcaA2p1     0x90e3170300cb373c  0x90e3170300cb373c  fit-nvos-vrt-mec-161-002 smi2  1               swA2p1
hcaA2p1pl1  0x90e3170300cb373c  0x90e3170300cb373c  fit-nvos-vrt-mec-161-002 smi2  1               swA2p1pl1
hcaA2p1pl2  0x90e3170300cb373c  0x90e3170300cb373c  fit-nvos-vrt-mec-161-002 smi2  2               swA2p1pl2
hcaA2p1pl3  0x90e3170300cb373c  0x90e3170300cb373c  fit-nvos-vrt-mec-161-002 smi2  3               swA2p1pl3
hcaA2p1pl4  0x90e3170300cb373c  0x90e3170300cb373c  fit-nvos-vrt-mec-161-002 smi2  4               swA2p1pl4
hcaA2p2     0x90e3170300cb3f3c  0x90e3170300cb3f3c  fit-nvos-vrt-mec-161-003 smi0  1               swA2p2
hcaA2p2pl1  0x90e3170300cb3f3c  0x90e3170300cb3f3c  fit-nvos-vrt-mec-161-003 smi0  1               swA2p2pl1
hcaA2p2pl2  0x90e3170300cb3f3c  0x90e3170300cb3f3c  fit-nvos-vrt-mec-161-003 smi0  2               swA2p2pl2
hcaA2p2pl3  0x90e3170300cb3f3c  0x90e3170300cb3f3c  fit-nvos-vrt-mec-161-003 smi0  3               swA2p2pl3
hcaA2p2pl4  0x90e3170300cb3f3c  0x90e3170300cb3f3c  fit-nvos-vrt-mec-161-003 smi0  4               swA2p2pl4

Viewing a Specific HCA Port

To inspect a specific HCA port, run the following command:

admin@nvos:~$ nv show peer-port hcaA2p1

To view counters, run the following command:

admin@nvos:~$ nv show peer-port hcaA2p1 counters

To view link information, run the following command:

admin@nvos:~$ nv show peer-port hcaA2p1 link

See the https://nvidia.atlassian.net/wiki/x/hEjc2Q section below for full details and output examples.

Limitations

  • When an HCA link transitions up or down, it may take up to 1 minute for the change to appear in nv show peer-port.

  • PHY lane statistics are reported in nv show peer-port <peer-port-id> link phy health.

HCA Telemetry Commands

Last updated: