Silent job performance degradation caused by Adaptive Routing (AR) hardware resource exhaustion went undetected until NVIDIA Minerva identified the root cause — reducing resolution time by approximately one to two weeks compared to similar issues in production.
Challenge
Jobs were completing below expected performance with no apparent cause. Standard checks passed and configuration appeared correct, leaving operators without a clear path to root cause identification.
Proactive Detection
NVIDIA Minerva automatically identified a failure to add neighbor table entries on multiple switches, revealing undetected AR hardware resource exhaustion that was silently degrading job performance.
Mitigation
The detection led to a critical bug filed with engineering. Root cause analysis confirmed that port flapping had caused the AR hardware resource exhaustion. Engineering was able to begin targeted triage immediately rather than starting from an uncharacterized performance complaint.
Operational Impact
-
Prevented extended job performance degradation.
-
Reduced Mean Time to Resolution (MTTR) by approximately one to two weeks compared to similar issues in production.
-
Enabled faster engineering triage and root cause identification.
-
Surfaced previously hidden infrastructure resource constraints.
Last updated: