Switch Health

Verifies that switch hardware sensors (temperature, fans, PSUs) report nominal values at idle and that the supervised log-classification library (Collective Learning) is not firing on categories that signal an impending failure. Switch-side issues are often subtle during the ORR: a switch can be functionally up while a supervised log category quietly accumulates events.

Persona Use Cases

Validation Procedure

Step 1: Confirm Switch Critical-issue Summary Tiles Are Clean

1. In the user-tenant, open AI Cluster Health.

2. Inspect Health Issues (Summary Tiles):

   • Switch Critical Issues: count of high-priority system and fabric alerts detected by Collective Learning. Should be 0.

   • Server Critical Issues: same, for hosts (validated alongside switches because the supervised library is shared).

Step 2: Walk the Collective Learning Heatmap

1. Inspect Log Errors > Switch, GPU and Server Critical Issues grid view. The Collective Learning library categorizes events including:

   • Out of Memory Processes Killed

   • Network Link Disconnections or Instability

   • GPU Memory Errors

   • Firmware/OS Driver Mismatches

   • Thermal Throttling

   • PCI Advanced Error Reporting

   • Storage Client Evictions

2. Walk every category. Each should be empty or near-empty in the idle window.

3. Hot spots in the heatmap (high-density cells) localize whether the issue is single-node, single-rack, or systemic across racks.

Step 3: Validate Switch Sensors Are Reporting

1. Confirm that switch sensor metrics (temperature, fan speed, PSU status) are present. These metrics typically arrive through NMX-T or vendor OTLP. No preconfigured dashboard has a dedicated tile for them, so use the Trends page to plot:

   • Switch chassis temperature over the validation window: should be flat and within vendor spec.

   • PSU input/output power: both PSUs reporting on dual-PSU chassis.

   • Fan RPM across all fan trays.

2. Spot-check three to five switches across leaf, spine, and super-spine tiers.

Step 4: Cross-check Switch Syslog Ingestion

1. On Telemetry Health, confirm that the Switch Syslog count is non-zero. A non-zero count shows that switches are shipping syslog, even if no events have fired.

2. A Switch Syslog of zero with all other counters non-zero usually indicates the syslog forwarder on the switches is not configured, not that switches are healthy.

Step 5: Spot-check Daily Check Logs

1. On Job Failure Root Cause Analysis > Daily Check Logs, check the Collective Learning categories for hosts, such as smartd, nhc (Node Health Check), and dhclient. The panel shows these even when no jobs are running. Confirm that they are quiet.

2. The presence of nhc failures at idle indicates nodes that the scheduler will likely refuse to allocate: escalate to the workload-ops team before the ORR sign-off.

Reference Dashboards and Sections

• AI Cluster Health: Health Issues (Switch Critical Issues, Server Critical Issues), Log Errors, Metric Errors (Switch PHY/Symbol Errors).

• Job Failure Root Cause Analysis: Daily Check Logs for smartd, nhc, and dhclient.

• Telemetry Health: Device Syslog, Switch Syslog, Node Syslog counters.

• Trends: switch sensor metric plots until a dedicated switch-sensors dashboard ships.

Pass Criteria

• [ ] Switch Critical Issues and Server Critical Issues summary tiles read 0.

• [ ] No Collective Learning category is densely populated on the Log Errors heatmap.

• [ ] Switch chassis temperatures are flat and within vendor spec across the validation window.

• [ ] PSU and fan-tray status is nominal on spot-checked switches.

• [ ] The Switch Syslog counter on Telemetry Health is non-zero.

• [ ] Daily Check Logs (smartd, nhc, dhclient) are quiet at idle.

• [ ] Idle-baseline screenshots of AI Cluster Health Log Errors and Telemetry Health archived.

Common Pitfalls

• Switch Syslog is zero but switches are clearly forwarding syslog elsewhere. Either the collector is configured for the wrong destination, or the syslog priority filter on the switch is dropping events before they leave. Validate with tcpdump on the collector host if needed.

• Collective Learning category empty because the library doesn't recognize the message. Collective Learning is supervised: it only categorizes patterns it has been trained on. A switch can be emitting genuinely problematic syslog that lands in the "uncategorized" bucket and shows up only on AI Cluster Health > Zero Day Log Anomalies. Cross-check there as part of switch-health sign-off.

• Sensor data missing for one switch model. Sensor-data export depends on the on-switch agent (NMX-T, gNMI, or vendor OTLP); a switch model not in the library may be functionally up but not reporting sensors. Escalate to the integration team.

• Idle "Thermal Throttling" hits. This almost always traces to under-spec cooling or in-rack airflow problems rather than the switch itself. DC Ops has the right context.

Last updated: October 21, 2026