Job Validation

Verifies that the Slurm scheduler is integrated end to end with NVIDIA Minerva: events ingest, jobs appear in the dashboards, the Job FSM transitions through the expected states, and the Job ID filter scopes the rest of the dashboards as designed. This is the final ORR validation: it confirms that the operational platform is ready to support real workloads and to support root cause analysis (RCA) when failures happen.

Persona Use Cases

Validation Procedure

Step 1: Confirm Slurm Ingestion via Telemetry Health

1. In the user-tenant, open Telemetry Health.

2. Confirm:

   • SLURM Records: non-zero. Tracks ingestion of job and node events from the workload scheduler.

   • SLURM Node Event: non-zero (or zero only if no node-state changes have occurred since install).

3. A zero SLURM Records count means the Slurm event collector or Slurm log path (Slurm) isn't shipping data. Resolve this before continuing.

Step 2: Validate Slurm Node-state Surfaces

1. Open the Professor-X Dashboard > SLURM:

   • Slurm Node State: real-time count of nodes in the Allocated, Idle, Drained, and Down states.

   • Drained/Down Nodes: summary tiles flagging nodes removed from the scheduling pool.

2. Confirm the counts match the expected post-install state. A typical Day 0 cluster has all nodes in Idle, apart from any small subset in pre-allocated test partitions.

3. Investigate any unexpected Drained or Down nodes before submitting the reference jobs: they may be excluded for hardware reasons that GPU Health, NVLink Health, or PHY Errors have already surfaced.

Step 3: Submit a Reference Job

Coordinate with the workload-ops team to submit a small, controlled job:

1. Submit a single-node srun --gres=gpu:1 job that exercises a GPU briefly (for example, an nvidia-smi loop or a 30-second gpuburn-style probe). This validates the simplest path end to end.

2. Submit a small multi-node job (two to four nodes, one GPU per node) that exercises one NCCL collective. This validates the multi-node path including NVLink and fabric correlation.

Step 4: Validate the Job in Job Failure Root Cause Analysis

1. Open Job Failure Root Cause Analysis.

2. Inspect the Full Stack Job RCA > Job Status table. Confirm:

   • jobIdList, jobName, and nodes columns are populated for the reference job.

   • status transitions through RUNNING and ends at COMPLETED.

3. Confirm that the Failed vs Completed Jobs summary tiles increment correctly: Completed Jobs should increase by the number of reference jobs run, and Failed Jobs should not change.

4. Use the Job ID Filter to add the reference job's jobIdList to the global filter at the top of the dashboard. Confirm the entire dashboard re-scopes to that single job's events.

Step 5: Validate Per-job Filtering on AI Cluster Health and Professor-X

1. With the same jobIdList filter applied, open AI Cluster Health and Professor-X Dashboard. Both should re-scope correctly.

2. The reference job should show modest GPU utilization on its assigned nodes during its run window.

Step 6: Confirm the FSM Transitioned Cleanly

1. The Job FSM consumes scheduler events from the eventNetwork ingest path and builds an in-memory topology of the job. For each reference job, confirm in Trends or the Job RCA detail view that the lifecycle transitioned JobStart → RUNNING → JobEnd/COMPLETED without a stuck-state gap.

2. Known limitation: the Job FSM transition table does not currently allow a canceled job to restart on a STARTED event (a canceled job cannot transition back to RUNNING). This affects how operators read post-mortem job timelines: it is a finite-state-machine constraint, not a data-loss issue.

Step 7: Confirm Daily Check Logs Are Quiet for Involved Nodes

1. On Job RCA > Daily Check Logs, confirm smartd, nhc, and dhclient are clean for the nodes that ran the reference jobs. A failing nhc after job completion is a sign that the job stressed something the health check picked up.

Reference Dashboards and Sections

• Job Failure Root Cause Analysis: Full Stack Job RCA, Zero Day Log Anomalies, Log Errors Supervised, Metric Errors, Daily Check Logs.

• Telemetry Health: SLURM Records, SLURM Node Event counters.

• Professor-X Dashboard: SLURM.

• AI Cluster Health: used with the jobIdList filter to validate per-job event scoping.

Pass Criteria

• [ ] SLURM Records counter on Telemetry Health is non-zero.

• [ ] Slurm Node State on Professor-X Dashboard correctly reflects the post-install state of every node.

• [ ] No unexplained Drained or Down nodes are present.

• [ ] Single-node reference job submitted, observed in the Job Status table, and ended in the COMPLETED state.

• [ ] Multi-node reference job submitted and ended in the COMPLETED state.

• [ ] Failed vs Completed Jobs tiles incremented as expected.

• [ ] Job ID Filter re-scopes Job RCA, AI Cluster Health, and Professor-X Dashboard correctly.

• [ ] Job FSM transitioned JobStart → RUNNING → COMPLETED cleanly for both reference jobs.

• [ ] Daily Check Logs (smartd, nhc, dhclient) remained quiet for the nodes that ran the reference jobs.

• [ ] Idle-baseline screenshots of Job RCA, Professor-X Dashboard SLURM, and Telemetry Health archived.

Common Pitfalls

• SLURM Records is non-zero but SLURM Node Event is zero. Job records are flowing but node-state events are not. This usually traces to a separate Slurm event collector configuration: both paths need to be onboarded as described in Slurm.

• Reference job COMPLETED but does not appear in Job RCA. The dashboard's time filter may exclude the job's window; re-scope to a window that covers submission through completion.

• jobIdList filter does not narrow other dashboards. The dashboard you're filtering may not include jobIdList as a global-filter dimension. Add it in the dashboard's filter pane configuration (see Dashboard > Build Dashboards).

• Confusion over canceled-job restarts. Per the FSM limitation in Step 6, a canceled job is terminal in the FSM. If the operator cancels and resubmits, the resubmission will appear as a new jobIdList value: there is no state-table path from CANCEL back to RUNNING for the same job ID. Document this behavior in the team's runbook so post-mortems are read correctly.

• nhc failure after the reference job. This may indicate the reference job exercised something fragile (storage path, network file system mount). Investigate it before declaring the cluster ready for production workloads.

Last updated: October 21, 2026