When a switch fails, the collection shows an error at the top with the switch name, a message, and an error code, for example [sim-switch-02] Switch NVUE error: the switch reported its HFT session failed and nvue_session_failed. The switch's session and ports show Failed; click the info icon next to the session for details. Codes that start with nvue_ come from the switch, codes that start with collector_ come from the collector, and the other codes come from the platform.
|
Error code |
Cause |
What to do |
|---|---|---|
|
|
The switch does not export HFT to this installation: the destination, port, or stats group is wrong or missing, or the destination is an IP address instead of |
Configure telemetry export. See Prepare Switches for HFT. |
|
|
The |
Run |
|
|
Another HFT session is running on the switch |
Wait for it to end, or stop it. |
|
|
The switch rejected the stored credentials |
Import the correct credentials. See Register Switch Credentials for HFT. |
|
|
The switch rejected a setting in the collection |
Read the message, and correct the collection. |
|
|
The switch could not configure or apply HFT |
Read the message, and check |
|
|
The switch didn't finish applying in time |
Check the switch for a pending revision. See Clear Pending Revisions. |
|
|
The switch's HFT session failed during the collection |
Check the HFT session state on the switch. See Check HFT on a Switch. |
|
|
The credentials ask for certificate verification, and the switch does not present the NVIDIA Minerva certificate |
Install the certificate, or import the credentials with |
|
|
The collector can't reach the switch's NVUE API |
Check the address and port in the credentials, and that the collector can reach the switch on that port. |
|
|
The collector did not answer the platform in time |
Check that the collector is running and connected to the platform. |
|
|
The switch isn't managed by its assigned collector |
Check the switch's collector in the topology. |
|
|
The collector could not complete the request |
Retry. If the error continues, collect the collector logs and contact NVIDIA support. |
|
|
The switch didn't start within 10 minutes, or before the duration ended |
Check the collector and the switch, and then start a new collection. |
|
|
The collection was stopped before the switch finished configuring |
Nothing; NVIDIA Minerva has already cleaned up the switch. |
|
|
The collection expired before configuration began |
Start a new collection. |
A Collection Finishes but Has No Data
-
Check the switch's HFT destination: port 4321, stats group
ONLY-HFT, and a.nip.iohost name. -
With remote collectors, check that the switch can reach the collector on port 4321, and that the collector can reach the platform on port 14321.
-
For traffic-class, switch-priority, or priority-group counters, check that the collection also included the classes, priorities, or groups.
HFT Timestamps Are 37 Seconds Early
HFT samples from switches running Cumulus Linux 5.19 or later are shown 37 seconds earlier than they were taken. On Cumulus Linux 5.18, switches export HFT timestamps 37 seconds ahead of UTC, and NVIDIA Minerva corrects for that by default. Cumulus Linux 5.19 fixes the timestamps on the switch (Cumulus Linux issue 5235587), so the default correction in NVIDIA Minerva shifts them a second time. This affects HFT queries, the Trends and Explore views, and the 10-second HFT summaries.
If every switch you collect HFT from runs Cumulus Linux 5.19 or later, change the correction:
-
On the platform host, as the installation user, open
~/opt/minerva/etc/hft-db.json. -
Under
profiles>cumulusLinux, delete thekeyAttrsentry, both its name and its list. WhilekeyAttrsis in the file, NVIDIA Minerva ignores every change to the file. Do not change anything else. HFT keeps using its built-in key attributes, so existing HFT series aren't affected. -
In the same
cumulusLinuxsection, setutcOffsetNanosto0. -
Check that the file is still valid JSON:
python3 -m json.tool ~/opt/minerva/etc/hft-db.json > /dev/null && echo OK -
Within a minute, confirm that HFT is using the new value:
grep utcOffsetNanos ~/opt/minerva/var/hft-db/hft-db.json.runIf it still shows the old value, check steps 2 and 3.
The change applies within seconds, without a restart, and it's kept across restarts. Stored HFT samples are shown at the corrected time right away. 10-second HFT summaries written before the change keep the 37-second shift, and about 37 seconds of summaries around the moment of the change are missing.
To tell which setting your switches need, compare their HFT data with their regular telemetry:
|
HFT data is |
Set |
|---|---|
|
37 seconds ahead of regular telemetry |
|
|
37 seconds behind regular telemetry |
|
If you raise the value back to 37000000000, about 37 seconds of summaries around the change are counted twice.
If you collect HFT from a mix of Cumulus Linux 5.18 and 5.19 or later switches, no single setting is correct for all of them: the timestamps from one group stay 37 seconds off.
Check HFT on a Switch
To see the switch's HFT configuration and the status of its last HFT session, run nv show system telemetry hft on the switch. After a collection ends, export state is disabled and session-status shows the final status, for example COMPLETED.
To check the ONLY-HFT stats group, run nv show system telemetry stats-group ONLY-HFT hft. Its export state is disabled except while a collection is running.
A Collection Is Refused for Capacity
The collection would go over a capacity limit. Make it smaller: use fewer ports or counters, or a longer sampling interval. You can also wait for other collections to end. An administrator can raise the limits. See Set HFT Capacity Limits.
Last updated: