Networking ConnectX-8 Firmware Release Notes

Known Issues

VF Network Function Limitations in SR-IOV Legacy Mode & in Switchdev Mode

Dual Port Device

Single Port Device

127 VF per PF (254 functions)

127

 VF+SF Network Function Limitations in Switchdev Mode

Dual Port Device

Single Port Device

  • 127 VF per PF (254 functions)

  • 512 PF+VF+SF per PF (1024 functions)

  • 127 VF (127 functions)

  • 512 PF+VF+SF per PF (512 functions)

Internal ref.

Issue

5236199

Description: Due to a limitation in module telemetry retrieval, pre-FEC BER and related telemetry values remain at zero and do not transition to live telemetry status. Other telemetry defined by the CMIS VDM specification is expected to function normally.

Workaround: N/A

Keywords: Telemetry retrieval, pre-FEC BER

Detected in version:

40.51.1002

5329699 / 5388950

Description: Setting adp_retx_profile_id to 0 through ROCE_ACCL after a user profile has been configured is silently ignored.

Workaround: N/A

Keywords: ROCE_ACCL, adp_retx_profile_id

Detected in version:

40.51.1002

301219 / 5224356

Description: On GB200 setups with DDP enabled, the HPC-X application hangs and prints the warning "got active message id 0, but no handler installed".

Workaround: Set the environment variable UCX_IB_ODP_MEM_TYPES=host.

Keywords: GB200, DDP, HPC-X, UCX, hang

Detected in version:

40.51.1002

5043315

Description: When running bidirectional ib_write_bw traffic on both interfaces simultaneously, low performance may be observed due to high PCIe latency (~4 µs) in certain system configurations.

Workaround: Decrease GABL to 192 KB (default is 282 KB for this setup) using mlxconfig to improve bidirectional bandwidth to approximately 1,450 Gb/s.

Keywords: ib_write_bw, PCIe latency, GABL, mlxconfig, bidirectional bandwidth

Detected in version:

40.51.1002

4893565 / 4532684

Description: Occasional high spikes in operation latency may be observed.

Workaround: N/A

Keywords: latency spike, operation latency

Detected in version: 

40.51.1002

4759910 / 5011023

Description: During an ISSU or Live Firmware Patch over InfiniBand, the NIC may be temporarily unreachable by the OpenSM subnet manager for the duration of the ISSU process, which can result in "Failed to get sminfo" errors.

Workaround: N/A

Keywords: ISSU, LFWP, InfiniBand, OpenSM, sminfo

Detected in version: 

40.51.1002

4705948 / 4684233

Description: A bandwidth degradation of approximately 25% may be observed when performing DC RDMA Read operations between ConnectX-7 (NDR) and ConnectX-8 using a low number of QPs (1–16). The degradation diminishes as QP count increases.

Workaround: N/A

Keywords: DC RDMA Read, bandwidth degradation, NDR, QP

Detected in version: 

40.51.1002

4705241 / 4684233

Description: When Quantum-2 is part of an XDR topology, serving as a leaf switch connected to NDR-based hosts, a bandwidth degradation of approximately 3–7 Gb/s is expected.

Workaround: N/A

Keywords: Quantum-2, XDR, NDR, leaf switch, bandwidth degradation

Detected in version: 

40.51.1002

4628696

Description: When HASH LAG single QP is enabled, ib_read_bw for a single QP over hash LAG can reach up to 337 Gbps, while ib_write_bw for a single QP can achieve up to 390 Gbps.

Workaround: N/A

Keywords: HASH LAG, single QP, ib_read_bw, 337 Gbps

Detected in version: 

40.51.1002

4436843

Description: When the link recovery feature is enabled with a link-down timeout of 0, port recovery counters are unexpectedly reset.

Workaround: N/A

Keywords: link recovery, PPRM, recovery counter, link-down timeout

Detected in version: 

40.51.1002

Internal Ref.

Issue

4970263 / 4989004

Description: A high-temperature assert may be reported only the first time the chip reaches the thermal threshold.

Workaround: N/A

Keywords: Temperature

Detected in version:

40.49.1014

5085252 / 5090080

Description: Following a hot reset, a false Surprise Link Down (SLD) alarm may be reported intermittently on the Downstream Port (DSP) through the Advanced Error Reporting (AER) register. This is a false alarm and has no functional impact on link stability or data integrity.

Workaround: N/A

Keywords: Surprise Link Down (SLD)

Detected in version:

40.49.1014

4901395 / 4919262

Description: When PXE is enabled for both Legacy (FlexBoot) and UEFI drivers, the UEFI driver supports only a single TFTP session after the DHCP handshake.

Workaround: For scenarios that require multiple concurrent TFTP sessions, disable PXE filters via the HCA HII menu.

Keywords: PXE, Filter enabled, UEFI

Detected in version: 40.49.1014

4893639

Description: Direct Memory Access (DMA) protection is not supported in the ipxe.efi driver.

Workaround: N/A

Keywords: DMA

Detected in version: 40.48.1000

4594515 / 4622127 / NVbug 5460310

Description: Running perftest with ib_send_bw using a message size of 4KB and 1000 QPs does not reach the expected full line rate.

Workaround: N/A

Keywords: perftest

Detected in version: 40.48.1000

4604969

Description: Probe packets might be dropped at the transmission stage when multiple congestion control flows are active.

Workaround: N/A

Keywords: PCC, RTT, probe

Detected in version: 40.47.1026

4496642

Description: The timestamps (t2, t4) of the received RTT probes are taken from the free-running clock, even when ROCE_CC_RTT_TIMESTAMP_FORMAT is set to 0x02.
The format of all RTT probe timestamps can be found in HCA_CAP.rtt_timestamp_format.

Workaround: N/A

Keywords: RTT RTC timestamp

Detected in version: 40.47.1026

4705241

Description: When Quantum-2 is part of an XDR topology, serving as a leaf switch connected to NDR-based hosts, a bandwidth degradation of approximately 3–7 Gb/s is expected.

Workaround: N/A

Keywords: XDR, NDR, Quantum-2

Detected in version: 40.47.1026

4705948

Description: When using DC as the InfiniBand transport type to perform an ib_read RDMA operation between ConnectX-7 (NDR) and ConnectX-8, a bandwidth degradation of approximately 25% may be observed when using a low number of QPs (1–16). The performance degradation diminishes as the number of QPs increases.

Workaround: N/A

Keywords: DC, ib_read RDMA, NDR, performance

Detected in version: 40.47.1026

4685736

Description: Creating a DPA process that allocates a 128 MB data segment and loads a dynamic library may fail with syndrome 0xdc30ac.

Workaround: Limit the DPA application’s data segment size to 64 MB.

Keywords: DPA

Detected in version: 40.47.1026

4618452

Description: When only a partial number of fiber lanes are connected to a module (i.e., not all supported lanes are populated), the module re-insertion process may experience an extended link-up time of approximately one minute. This occurs because the module requires additional time to detect that only a subset of the lanes is connected.

Workaround: N/A

Keywords: Cables

Detected in version: 40.47.1026

4270913

Description: For DC InfiniBand transport, using SRQs requires distributing QPs across four SRQs. Each QP must be assigned to its own SRQ when there are up to four QPs.

Workaround: If more than four QPs are used, they must be evenly distributed across the four available SRQs.

Keywords: DC InfiniBand transport, SRQ, QPs

Detected in version: 40.47.1026

4665802

Description: Due to a re-burst scheduler limitation, the expected “speed of light” value for single QP RDMA Write on ConnectX-8 devices cannot be calculated. Currently, verification is limited to identifying performance degradations relative to previous firmware versions.

Workaround: N/A

Keywords: Single QP RDMA Write

Detected in version: 40.47.1026

4394475

Description: The existing congestion control configuration applies globally, rather than on a per-priority basis.

Workaround: Ensure that the configuration values for all priorities are aligned in either mlxconfig ROCE_CC_PRIO_MASK_P$port or sysfs ecn/roce_rp/enable/$port.

Keywords: Congestion control, ROCE_CC_PRIO

Detected in version: 40.45.1020

4201405

Description: Upgrading to firmware 40.44.0xxx from any previous Engineering Sample version requires power cycling the driver and not just resetting it (using mlxfwreset).  

Workaround: N/A

Keywords: Upgrade, power cycle, reset

Detected in version: 40.44.0208

Last updated: