Use this section when a triage section has produced evidence, but the issue is unresolved, needs cross-team review, or must refer to support, engineering, or a product specialist.
The escalation package lets the next owner start from your evidence without repeating the investigation.
Quick Decision
|
Symptom |
Action |
|---|---|
|
A reproducible failure with complete evidence |
Escalate with the branch-specific package. |
|
An intermittent failure with Good-State and Bad-State evidence |
Escalate with the state comparison and incident timing. |
|
A suspected hardware, link, PCIe, or transport fault |
Escalate with the relevant hardware or transport branch evidence. |
|
A workload-only or scale-only failure |
Attach the workload evidence from RDMA, Application, and NCCL Workload Triage and the comparison from Performance, Topology, and GPUDirect Triage. |
|
A gap in the evidence |
Collect only the missing item, do not restart the triage. For collection steps, see Common Intake and Evidence Collection. |
Escalation Summary
Start every escalation with a summary that covers these fields.
|
Field |
What to include |
|---|---|
|
Impact |
What is unavailable, degraded, or blocked |
|
Scope |
Affected hosts, ports, rails, nodes, users, or workloads |
|
Symptom |
Exact error, observed behavior, and first and last occurrence |
|
Reproduction |
Smallest reproducer, reproduction rate, and the first scale that fails, if relevant |
|
Fault domain Conclusion |
The section your evidence points to, and why |
|
Changes Already Made |
Recovery actions, controlled comparisons, and their results |
State what you know, what you have ruled out, and the question or action you need from the next owner.
Required Attachments
|
section |
Required evidence |
|---|---|
|
Device Visibility and PCIe |
Host bundle, expected and actual PCIe state, host and platform events, controlled comparison |
|
Physical Link |
Host bundle, |
|
Firmware and Driver |
Current and known-good host bundles, approved baseline, change history, activation result |
|
InfiniBand Fabric |
Host bundle, topology and port mapping, SM or UFM package, |
|
RoCE Transport |
Host bundles from both endpoints, path and traffic-class design, switch evidence, controlled endpoint test |
|
Workload and NCCL |
Exact command, NCCL log for each rank, mapping, minimal test result, transport evidence |
|
Performance and Topology |
Test configuration, expected and observed distribution, GPU, NIC, PCIe, and NUMA mapping, smallest reproducer |
Escalation Quality Check
Before you hand off the case, confirm the following:
-
The incident time is recorded, and all evidence covers that window.
-
The affected path is mapped end to end.
-
The known-good comparison is included, or the package states that none is available.
-
The evidence is attached to the case.
-
Sensitive data, such as hostnames, IP addresses, and credentials, is removed, unless the receiving process permits it. [TBD: link the approved procedure for removing sensitive data from host bundles.]
-
The request states the decision or action needed from the next owner.
Close or Continue
Close the incident only when the symptom is resolved and you've verified the result with the original workload or an equivalent controlled test.
If service is restored but the cause isn't known, keep the evidence and mark the case as recovered with root cause pending.
Last updated: