Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

Job Failure Root Cause Analysis

The Job Failure Root Cause Analysis dashboard is used for diagnosing the root cause of AI job failures. By overlaying job scheduler data with full-stack telemetry — ranging from GPU-specific XIDs to network-wide pause frames — you can determine whether a failure was caused by application code, hardware defects, or fabric congestion.

Persona Use Cases

Persona

Primary Use Case

AI/ML Engineers

Correlate job FAILED or CANCELLED states with GPU XID errors to separate software bugs from hardware-induced crashes.

System Administrators

Monitor the Daily Check Logs to find background system failures (for example, smartd or nhc) that may have caused a node to be drained mid-job.

Network Engineers

Analyze RoCE Adaptive Retransmits and Pause Frames to identify straggler nodes impacting collective communication performance during a training run.

Dashboard Overview

The Job Failure Root Cause Analysis dashboard is the primary dashboard for investigating slow or failing jobs.

Full Stack Job RCA

This section provides the high-level context for the investigation.

  • Job Status: Displays critical metadata from the scheduler (Slurm), including jobIdList, jobName, and the specific nodes assigned to the job. It tracks job status such as COMPLETED, FAILED, or TIMEOUT.

  • Failed vs. Completed Jobs: Summary tiles provide a quick count of the overall cluster job success rate within the selected timeframe.

  • Job ID Filter: Filters the entire dashboard to issues and events associated with a given job by adding jobIdList to the global filters at the top of the dashboard.

Zero Day Log Anomalies

This section uses NVIDIA Minerva's unsupervised machine learning to surface rare or unknown log patterns.

  • Identifies logs that have not been seen recently across the cluster.

  • Example: Detecting LNetError: peer has no available nets, which points to a Lustre networking issue that could prevent compute nodes from accessing training data.

  • Every anomaly includes an AI-generated guide with specific terminal commands (for example, lctl list_nids) to resolve the issue.

Log Errors (Supervised)

These widgets track known critical error signatures categorized by the NVIDIA Minerva Collective Learning library.

  • Host XID Logs: A detailed table of GPU XID errors. Captures high-impact events like XID 31 (GPU memory page fault) and XID 137 (NVLink PRIV error).

  • NVSwitch SXid Logs: Tracks fabric-level errors, such as SXid 12028, identifying non-posted PRIV errors on specific NVSwitch links.

  • Switch and Server Critical Issues: A heatmap that identifies hot spots of errors across the hostnames and switches involved in the job.

Metric Errors

Analyzes physical and logical link health across the fabric.

  • Switch Interface Physical Layer Uncorrectable Errors: Monitors uncorrectableErrors on switch ports. A high count indicates signal integrity issues or failing optics that FEC cannot resolve.

  • Switch Symbol Errors: Visualizes layer2SymbolErrors over time, used to identify degrading cables before they cause a link-down event.

Health Issues (Summary Tiles)

Real-time counters for high-priority hardware alerts.

  • GPU XID Errors: Aggregate count of all XIDs detected during the job window.

  • Anomalous Servers/Switches: Counts how many unique devices are reporting statistically significant deviations.

  • Link Flaps: Tracks interruptions in the network connectivity of compute nodes.

Daily Check Logs

Monitors secondary system indicators that often signal an unstable node.

  • smartd: Reports increases in NVMe drive error log entries.

  • nhc (Node Health Check): Reports script timeouts or failures (for example, check_cmd_output timeouts) that might cause the scheduler to exclude a node.

  • dhclient: Monitors packet failures over management interfaces (for example, enp26s0f0v1).

Key Metrics

  • GPU Temperature High: Flags any GPU exceeding 75°C.

Last updated: