Networking Minerva User Guide

Troubleshoot HFT

When a switch fails, the collection shows an error at the top with the switch name, a message, and an error code, for example [sim-switch-02] Switch NVUE error: the switch reported its HFT session failed and nvue_session_failed. The switch's session and ports show Failed; click the info icon next to the session for details. Codes that start with nvue_ come from the switch, codes that start with collector_ come from the collector, and the other codes come from the platform.

Error code

Cause

What to do

nvue_otlp_destination

The switch does not export HFT to this installation: the destination, port, or stats group is wrong or missing, or the destination is an IP address instead of .nip.io

Configure telemetry export. See Prepare Switches for HFT.

nvue_stats_group

The ONLY-HFT stats group doesn't exist on the switch

Run nv set system telemetry stats-group ONLY-HFT hft export state disabled and nv config apply.

nvue_session_active

Another HFT session is running on the switch

Wait for it to end, or stop it.

nvue_auth_failed

The switch rejected the stored credentials

Import the correct credentials. See Register Switch Credentials for HFT.

nvue_invalid_config

The switch rejected a setting in the collection

Read the message, and correct the collection.

nvue_configure_error, nvue_apply_error

The switch could not configure or apply HFT

Read the message, and check nv config history on the switch.

nvue_apply_timeout

The switch didn't finish applying in time

Check the switch for a pending revision. See Clear Pending Revisions.

nvue_session_failed

The switch's HFT session failed during the collection

Check the HFT session state on the switch. See Check HFT on a Switch.

nvue_cert_untrusted

The credentials ask for certificate verification, and the switch does not present the NVIDIA Minerva certificate

Install the certificate, or import the credentials with skip_verify empty. See Verify Switch Certificates for HFT.

collector_switch_unreachable

The collector can't reach the switch's NVUE API

Check the address and port in the credentials, and that the collector can reach the switch on that port.

collector_unavailable

The collector did not answer the platform in time

Check that the collector is running and connected to the platform.

collector_switch_not_managed

The switch isn't managed by its assigned collector

Check the switch's collector in the topology.

collector_error

The collector could not complete the request

Retry. If the error continues, collect the collector logs and contact NVIDIA support.

session_start_timeout

The switch didn't start within 10 minutes, or before the duration ended

Check the collector and the switch, and then start a new collection.

capture_cancelled

The collection was stopped before the switch finished configuring

Nothing; NVIDIA Minerva has already cleaned up the switch.

capture_expired

The collection expired before configuration began

Start a new collection.

A Collection Finishes but Has No Data

  • Check the switch's HFT destination: port 4321, stats group ONLY-HFT, and a .nip.io host name.

  • With remote collectors, check that the switch can reach the collector on port 4321, and that the collector can reach the platform on port 14321.

  • For traffic-class, switch-priority, or priority-group counters, check that the collection also included the classes, priorities, or groups.

HFT Timestamps Are 37 Seconds Early

HFT samples from switches running Cumulus Linux 5.19 or later are shown 37 seconds earlier than they were taken. On Cumulus Linux 5.18, switches export HFT timestamps 37 seconds ahead of UTC, and NVIDIA Minerva corrects for that by default. Cumulus Linux 5.19 fixes the timestamps on the switch (Cumulus Linux issue 5235587), so the default correction in NVIDIA Minerva shifts them a second time. This affects HFT queries, the Trends and Explore views, and the 10-second HFT summaries.

If every switch you collect HFT from runs Cumulus Linux 5.19 or later, change the correction:

  1. On the platform host, as the installation user, open ~/opt/minerva/etc/hft-db.json.

  2. Under profiles > cumulusLinux, delete the keyAttrs entry, both its name and its list. While keyAttrs is in the file, NVIDIA Minerva ignores every change to the file. Do not change anything else. HFT keeps using its built-in key attributes, so existing HFT series aren't affected.

  3. In the same cumulusLinux section, set utcOffsetNanos to 0.

  4. Check that the file is still valid JSON:

    python3 -m json.tool ~/opt/minerva/etc/hft-db.json > /dev/null && echo OK
    
  5. Within a minute, confirm that HFT is using the new value:

    grep utcOffsetNanos ~/opt/minerva/var/hft-db/hft-db.json.run
    

    If it still shows the old value, check steps 2 and 3.

The change applies within seconds, without a restart, and it's kept across restarts. Stored HFT samples are shown at the corrected time right away. 10-second HFT summaries written before the change keep the 37-second shift, and about 37 seconds of summaries around the moment of the change are missing.

To tell which setting your switches need, compare their HFT data with their regular telemetry:

HFT data is

Set utcOffsetNanos to

37 seconds ahead of regular telemetry

37000000000 (the default, for Cumulus Linux 5.18)

37 seconds behind regular telemetry

0 (for Cumulus Linux 5.19 or later)

If you raise the value back to 37000000000, about 37 seconds of summaries around the change are counted twice.

If you collect HFT from a mix of Cumulus Linux 5.18 and 5.19 or later switches, no single setting is correct for all of them: the timestamps from one group stay 37 seconds off.

Check HFT on a Switch

To see the switch's HFT configuration and the status of its last HFT session, run nv show system telemetry hft on the switch. After a collection ends, export state is disabled and session-status shows the final status, for example COMPLETED.

To check the ONLY-HFT stats group, run nv show system telemetry stats-group ONLY-HFT hft. Its export state is disabled except while a collection is running.

A Collection Is Refused for Capacity

The collection would go over a capacity limit. Make it smaller: use fewer ports or counters, or a longer sampling interval. You can also wait for other collections to end. An administrator can raise the limits. See Set HFT Capacity Limits.

Last updated: