RoCE (RDMA over Converged Ethernet) enables high-bandwidth, low-latency GPU-to-GPU communication in AI Factory clusters. Because RoCE is lossless by design, it relies on Priority Flow Control (PFC) to signal congestion: when a leaf switch becomes congested, it sends pause frames upstream. Those pause frames can cascade into retransmit storms that stall NCCL collective operations and cause job timeouts.
Trace a Job Timeout in Topology
The Topology view is the primary tool for tracing a job timeout back to the specific switch or link responsible.
-
Open the Topology tab and enter the job ID as a filter (for example,
4335424).The view isolates the spine, leaf, and compute nodes that participated in this job. Unrelated cluster topology is hidden.
-
Scan the filtered topology for anomaly indicators. Amber or red highlights on a node indicate that NVIDIA Minerva detected an anomaly at that location during the job's run window.
-
Identify the anomaly type on each highlighted node. In a RoCE retransmit scenario, look for:
-
Anomalous PFC (pause frames): the switch is generating pause frames that throttle upstream traffic.
-
RoCE retransmits: the adapter or link is experiencing packet loss that triggers RDMA retransmission.
-
-
Trace the impact. If a single leaf switch shows both pause frame and RoCE retransmit anomalies, it is likely the root cause. A congested leaf throttles all compute nodes directly connected to it, causing NCCL all-reduce operations across the job to stall until the job times out.
Last updated: