During an active In-Service Software Upgrade (ISSU) on a network switch, NVIDIA Minerva detected a rare control plane failure that standard monitoring missed, enabling operators to halt the maintenance window before any jobs were affected.
Challenge
A switch control plane failure occurred during an ISSU to Cumulus Linux 5.12.1.1000. The failure was unexpected and would not have been detected through standard monitoring, leaving it to propagate silently into job failures or performance degradation.
Proactive Detection
NVIDIA Minerva detected the issue automatically:
-
ML and NLP analysis detected a rare log entry indicating a previously undetected switch control plane failure.
-
The failure was identified during ISSU upgrade activity on the affected switch.
Mitigation and Savings
-
Operators halted the upgrade and maintenance window immediately, preventing the failure from affecting running jobs.
-
Estimated 768 hours of job runtime savings and 24 hours of switch uptime savings.
-
NVIDIA Minerva's recommended remediation guided operators through the investigation, including specific module logs and
cl-supportdiagnostics.
Last updated: