Networking Minerva User Guide
Networking Minerva User Guide
Breadcrumbs

Validate BGP and Adaptive Routing

Verifies that BGP sessions are established between every expected peer pair and that Adaptive Routing is operating without hardware resource exhaustion. Validation uses the BGP Sessions table in Topology, the BGP Operations dashboard, Fabric Validation, and Log Insights. The Performance Degradation case study (Adaptive Routing hardware resource exhaustion) describes the kind of issue this validation is meant to catch.

This validation area is part of the AI Factory Operational Readiness Review. Before you begin, complete the steps in Prepare for Readiness Validation.

Persona Use Cases

Persona

Primary Use Case

Network Engineers

Confirms every BGP peering between leaf, spine, and super-spine switches is in the ESTABLISHED state.

System Administrators / SREs

Confirms that switch logs contain no Adaptive Routing No More Resources notices, which signal hardware key-value database or neighbor-table exhaustion.

Validation Procedure

Step 1: Validate BGP Session State

  1. In the user-tenant, go to Topology > Provision > Metadata > BGP Sessions (see BGP Sessions).

  2. Filter or sort by BGP State and confirm:

    • All expected leaf-to-spine BGP sessions are ESTABLISHED.

    • All expected spine-to-super-spine BGP sessions are ESTABLISHED.

    • Peer Device Name is filled in for every session. A blank value means the far end isn't reporting its side.

  3. Open Fabric Validation and confirm that no BGP Not Established violations appear in the validation window.

  4. If you imported the BGP Operations dashboard, confirm that BGP Downs (24h) and BGP Flaps (24h) are 0 and that Observed Non-Established BGP Sessions (24h) is empty.

Step 2: Spot-Check in Topology

  1. Open the Topology page.

  2. Pick a representative leaf switch and confirm that its BGP Peer links and other link adjacencies match the design.

  3. Repeat for one spine and one super-spine.

Step 3: Check for Adaptive Routing Hardware Exhaustion

The Adaptive Routing Hardware Resource Exhaustion case in Performance Degradation appears as the NOTICE ROUTER: Failed to add entry to HW module. err: No More Resources log message on Cumulus Linux switches. During the ORR, confirm that this message is absent.

  1. Open Log Insights (in the user-tenant).

  2. Search syslog content for No More Resources, AdaptiveRoutingNotice, or the Cumulus Linux ROUTER application.

  3. Confirm that the search returns no results for the idle window.

  4. If your deployment's Collective Learning library includes the AdaptiveRoutingNotice category, confirm that it shows no hits. This category uses supervised learning to recognize these messages.

Step 4: Confirm That No Routing-Process Errors Are Logged

  1. On AI Cluster Health > Log Errors, check the Collective Learning heatmap for router-process or routing-protocol categories.

  2. Confirm that no kernel-level routing alarms (for example, FIB rejection events) appear.

Reference Dashboards and Sections

  • BGP Sessions (Topology): BGP session state, peers, and VRFs for every switch.

  • BGP Operations: session health, BGP down and flap counts, and non-established sessions.

  • Fabric Validation: BGP Not Established violations.

  • Topology page: spot-check BGP Peer links between leaf, spine, and super-spine switches.

  • Log Insights (in the user-tenant): searches for No More Resources and Adaptive Routing notices.

  • AI Cluster Health > Log Errors: Collective Learning categories.

  • Performance Degradation: the canonical example of why this validation matters.

Pass Criteria

  • [ ] All expected BGP sessions are ESTABLISHED in the BGP Sessions table, with Peer Device Name filled in.

  • [ ] No BGP Not Established violations appear in Fabric Validation during the validation window.

  • [ ] No No More Resources or AdaptiveRoutingNotice log messages are present in Log Insights.

  • [ ] The AdaptiveRoutingNotice Collective Learning category (if present) shows no hits.

  • [ ] The spot-checked leaf, spine, and super-spine switches in Topology have the expected neighbor relationships.

  • [ ] Idle-baseline notes are archived, including the number of expected BGP sessions and an explicit sign-off that no Adaptive Routing exhaustion notices were found.

Common Pitfalls

  • Sessions with no peer device. BGP session modeling requires Cumulus Linux 5.18 or later on both ends and LLDP between unnumbered peers. If a switch runs an older release or LLDP is off, its sessions show a blank Peer Device Name and don't appear as BGP Peer links.

  • A missing BGP session is harder to spot than a down session. A peer that never came up at all produces no events. Step 1 relies on counting the sessions you expect: a session that's in the design but missing from the BGP Sessions table is the problem.

  • Adaptive Routing exhaustion is subtle. The No More Resources message can appear long before jobs are visibly affected. In the Performance Degradation case study, it took one to two weeks to reach production triage without this signal. The ORR is the right time to confirm that the message is absent at baseline.

  • Log Insights checks aren't saved automatically. Capture the Log Insights search queries in the baseline archive so later operators can rerun the same checks.

Last updated: