Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

Sign-Off and Handoff

The sign-off is the moment the operator team formally states "the cluster is ready for workloads." Until sign-off, the ORR is incomplete. After sign-off, any unresolved findings carry over with explicit owners.

Persona Use Cases

Persona

Primary Use Case

Cluster Administrator

Owns the consolidated sign-off and the handoff conversation with the workload-owner team.

Tenant Admins

Confirms tenant-level inventory and grouping are correct before workloads are scheduled against tenant boundaries.

System Administrators / SREs

Owns the gap list (anything that didn't pass) and tracks it to resolution.

Aggregated Pass Criteria

A successful ORR has every box below checked. Each item rolls up the Pass Criteria from one validation page.

Foundation

  • [ ] Pre-Validation Setup: preconfigured dashboards imported into both tenants; idle window agreed and documented; baseline-archive location established.

  • [ ] Topology: device, interface, and SU counts match the build sheet; tenant grouping correct.

Hardware Health

  • [ ] GPU Health: every GPU reporting via DCGM; idle thermals, power, and utilization within spec; no XID activity at idle.

  • [ ] NVLink Health: XID 149.4, XID 149.37, XID 145, and Phy Symbol Errors all clean; Bandwidth Loss Ratio at baseline.

  • [ ] PHY Errors: Raw BER flat at baseline; Effective Errors (Post-FEC) at zero; PLR Retransmissions quiet.

  • [ ] Switch Health: Switch Critical Issues and Server Critical Issues tiles at zero; Collective Learning categories quiet; switch sensors and syslog ingestion confirmed.

Fabric Posture

  • [ ] Link Health: all links operationally up across all tiers; FSM Link Down Events quiet.

  • [ ] Layer 3: all expected BGP sessions in Established state; no Adaptive Routing "No More Resources" notices.

  • [ ] E-W Fabric Health: Adaptive Retransmissions, CNP Handled, ECN Marked, TX/RX Pauses, and NIC Discards all at idle baseline; numeric idle values archived.

  • [ ] ECMP and Multiplane Balance: uniformly low traffic across members at idle; explicit acknowledgment that workload-time validation is a post-ORR dependency.

Workload Integration

  • [ ] Job Validation: Slurm ingestion confirmed; reference single-node and multi-node jobs completed; Job ID filter scopes the dashboards correctly.

Baseline Archive

  • [ ] Idle-baseline screenshots from every section archived to the agreed location.

  • [ ] Numeric idle baselines (E-W fabric counters, BER, PLR rates) recorded.

  • [ ] Idle window timestamps documented (start, end, duration).

Gap List

Any unchecked item from the aggregated pass criteria above is an ORR gap. For each gap, document:

Field

Description

Section

The validation section that surfaced the gap.

Finding

One-sentence description of what was not green.

Owner

Person or role accountable for resolution.

Post-ORR disposition

One of: block-workload, track-but-not-blocking, accept-and-monitor.

Target resolution

Date or "before workload N" milestone.

A gap with disposition block-workload means workloads should not run until it is resolved. The Cluster Administrator decides which gaps are blocking.

Sign-Off

The Cluster Administrator confirms by recording:

  • The cluster name and tenant identifiers.

  • The ORR validation window (<start> through <end>).

  • The list of operators who participated (with their personas).

  • Any blocking gaps (or "None") and their dispositions.

  • A pointer to the baseline archive.

  • The sign-off date and the signing operator.

The signed sign-off record goes into the same archive that holds the baseline screenshots.

What Carries Over After the ORR

The following activities pick up where the ORR ends. Each should be tracked separately from the ORR sign-off:

  • Anomaly detector enablement: decide which anomaly objects in anomaly-detection.json to set enable: true for this cluster (start with anomalyRareSyslogPerAppSev, anomalyXidRate, anomalyPauseFramesLeafRxBySU, anomalyAdaptiveRetransmitsRoCE, anomalySwitchRawBitErrorRate, anomalyDeviceSyslogRate).

  • Notification endpoint configuration: configure Notifications and assign endpoints to anomaly views.

  • Auto Correlation namespace and view binding: see Auto Correlation.

  • Preconfigured Alerts enablement: enable the GPU Temp High, NVLink BW Loss, Rare Log with Remediation, and Collective Learning alerts in Preconfigured Alerts.

  • Workload-time baselines: re-validate ECMP balance, fabric pressure, and GPU thermals under a representative workload.

  • Resolution of any non-blocking ORR gaps.

Steady-State Operations (Out of Scope for the ORR)

For completeness, items that are explicitly out of scope for the ORR:

  • Steady-state operations cadence (daily checks, weekly sweeps).

  • Long-horizon trend reviews (capacity, optic aging, drift).

  • Detector tuning driven by incident postmortems.

  • Tenant-level chargeback and reporting workflows.

Common Pitfalls

  • Signing off with unresolved blocking gaps. Resist pressure to sign off "with caveats" if the caveats are real. Workload owners will read the sign-off as "ready for production" regardless of how it was qualified verbally.

  • No baseline archive. A signed ORR with no archived baselines makes every later "is this normal?" question unanswerable. The baseline is part of the deliverable.

  • Enabling detectors or notifications before ORR sign-off. Enabling notifications or anomaly detectors during ORR validation creates noise that contaminates the idle baseline. Keep the boundary clean.

Last updated: