Do these steps once on each switch before you include it in a collection. With remote collectors, the switches send telemetry to their collector. In a single-node installation, the platform host is the collector, so the switches send telemetry to the platform. The steps below call that address <collector-ip>.
Synchronize Switch Clocks
Each switch timestamps HFT samples with its PTP hardware clock, and timestamps regular telemetry with its system clock. For HFT collections and the HFT views to work correctly, both clocks on every switch must be within 0.5 seconds of the correct time. The PTP hardware clock keeps a time base that is 37 seconds ahead of the system clock, and NVIDIA Minerva corrects for that difference by default. For switches running Cumulus Linux 5.19 or later, see HFT Timestamps Are 37 Seconds Early in Troubleshoot HFT.
HFT runs without PTP, but to keep both clocks within 0.5 seconds of the correct time, configure PTP as Cumulus Linux requires: each switch takes its time from a grandmaster on the fabric, and the grandmaster should trace to the same source as the NVIDIA Minerva hosts. Cumulus Linux then sets the system clock from the PTP hardware clock. Cumulus Linux does not support setting the system clock from PTP and NTP at the same time, so follow Precision Time Protocol - PTP in the Cumulus Linux 5.18 documentation, including disabling NTP.
Warning: A switch configured for PTP that cannot reach a grandmaster becomes its own grandmaster, and nothing keeps its hardware clock accurate. Its HFT timestamps drift, but the switch does not report a problem. NTP can't correct this, because NTP does not set the PTP hardware clock.
Configure Telemetry Export
Run these commands on the switch:
nv set system telemetry stats-group ONLY-HFT hft export state disabled
nv set system telemetry export otlp grpc destination <collector-ip> port 4317
nv set system telemetry export otlp grpc destination <collector-ip>.nip.io port 4321
nv set system telemetry export otlp grpc destination <collector-ip>.nip.io stats-group ONLY-HFT
nv set system telemetry export otlp grpc insecure enabled
nv config apply
-
Port 4317 carries regular telemetry. NVIDIA Minerva also uses it to learn which collector manages the switch.
-
Port 4321 carries HFT data. It must use the
ONLY-HFTstats group, which must exist before you reference it. Create the stats group with HFT export disabled. NVIDIA Minerva turns it on during a collection and off again when the collection ends. Set the port and the stats group in separate commands; NVUE rejects both in one command. -
The HFT destination must be exactly
<collector-ip>.nip.io; an IP address or any other host name is refused.<collector-ip>.nip.ioresolves to<collector-ip>through public DNS. A site without internet access can resolve that same name through its own DNS server.
To check the result, run nv show system telemetry export otlp grpc destination. You should see <collector-ip> on port 4317, and <collector-ip>.nip.io on port 4321 with stats group ONLY-HFT.
With remote collectors, wait a few minutes after you configure the switch. The switch must appear in the topology, bound to its collector, before you start a collection.
Break Out Ports
Cumulus Linux does not support HFT on ports that use eight lanes. On Spectrum-4 switches, swp1 through swp64 use all eight lanes, so break out the ports you want to collect from. HFT then collects from the breakout interfaces, for example swp1s0 through swp1s3. See High Frequency Telemetry in the Cumulus Linux 5.18 documentation.
nv set interface swp1 link breakout 4x
nv config apply
Clear Pending Revisions
A pending NVUE revision on the switch can block HFT. Before you start a collection, list the switch's revisions through the NVUE REST API, and look for any whose state is pending:
curl -sk --user <username>:<password> \
"https://<switch-ip>:8765/nvue_v1/revision?rev=operational"
Delete each pending revision:
curl -sk --user <username>:<password> -X DELETE \
"https://<switch-ip>:8765/nvue_v1/revision/<revision-id>"
Allow NVUE Access from the Collector
The collector configures HFT through the switch's NVUE REST API, on port 8765 by default, or the port in the switch's credentials. The collector must be able to reach that port on every HFT switch. See Outbound from Collector Nodes (to Switches) in Collector Installation.
Last updated: