Known Issues

-

Description: NGINX running with the daemon on mode is not supported. This mode causes NGINX to fork into the background, which interferes with the proper initialization of XLIO.

Workaround: Disable NGINX daemon mode by adding the following directive to the NGINX configuration file: daemon off;

Keywords: NGINX; daemon mode

Discovered in Version: 3.0.2

4213696

Description: A kernel crash occurs when running high-performance XLIO workloads on the NVIDIA GH200 Grace Hopper. The crash is due to a threshold bug in the ARM SMMU upstream kernel, causing system instability.

Workaround: Follow the steps below.

1. Apply this patch to your kernel source tree.

2. Rebuild the kernel using your build system.

3. Install the updated kernel image.

4. Reboot to apply the changes.

Keywords: GH200; kernel; patch; Ubuntu 22.04

Discovered in Version: 3.40.2

4219402

Description: TCP traffic may hang during retransmission when attempting to split a packet.

Workaround: Enable XLIO_TX_WRE_BATCHING=1 to avoid hangs. 

Keywords: TCP hang; retransmission; split

Discovered in Version: 3.40.2

4238452

Description: TCP traffic may hang on a Tx only blocking socket.

Workaround: Use a non-blocking socket to avoid potential hangs during transmission.

Keywords: TCP hang; blocking socket

Discovered in Version: 3.40.2

4213201

Description: XLIO send queue corruption occurs under high connection load.

Workaround: N/A

Keywords: send queue corruption

Discovered in Version: 3.40.2

3823191

Description: In scenarios with high transmission congestion and a large number of connections, if the application sends more data into the socket than the hardware can transmit, the overall throughput may fall below the maximum wire speed.

Workaround: Ensure data transfer stays within the hardware's bandwidth capacity to prevent Tx drops.

Keywords: Wire speed; TX congestion; high connection scale

Discovered in Version: 3.40.2

3795997

Description: When the aggregated data to be sent exceeds the TCP send window, XLIO will postpone the transmission (Tx operation) until all previously sent data is acknowledged by the remote side.

Workaround: Increase the TCP receive window size on the remote side.

Keywords: TSO; quick ack; delayed ack; TCP window

Discovered in Version: 3.31.2

3795997

Description:  When aggregated data to send is bigger than the TCP send window allows - XLIO will postpone TX operation unless all previously sent data is acked by the remote side.

Workaround: Increase TCP receive window size on the remote side.

Keywords: TSO, Quick ack, Delayed ack, TCP window

Discovered in Version: 3.30.5

3865579

Description:  For small packets, up to 205 bytes, the removal of the BlueFlame feature from XLIO causes latency degradation. Expect up to a 400 ns increase in latency.

Workaround: N/A

Keywords: latency, blue flame

Discovered in Version: 3.30.5

3654779

Description: In the current state of the library, the TCP keep-alive feature is partially supported.
The known limitations are:
1. The keep-alive internal parameters are as follows:
a. TCP_KEEPINTVL - 75,000ms or 75 seconds (1:15 minutes)
b. TCP_KEEPCNT - 9 (probes to send before timeout)
2. The following setsockopt options are unsupported:
a. setsockopt(sock, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval)); - Change the intervals between the keep-alive probes  not operational
b. setsockopt(sock, IPPROTO_TCP, TCP_KEEPCNT, &retries, sizeof(retries)); - Change the number of probes not operational
3. The following system configurations are disregarded:
`sysctl -w net.ipv4.tcp_keepalive_intvl=60`
`sysctl -w net.ipv4.tcp_keepalive_probes=3`
4. The contents of the following files are disregarded:
/proc/sys/net/ipv4/tcp_keepalive_intvl
/proc/sys/net/ipv4/tcp_keepalive_probes

Workaround: N/A

Keywords: keep-alive, TCP_KEEPALIVE

Discovered in Version: 3.10

3440429

Description: XLIO fails to create RFS rule and prints the following error:

Create RFS flow failed, Tag: Y, Flow: dst: X.X.X.X:X, src: X.X.X.X:X, proto: TCP, family: INET, Priority: 1, errno: 17 - File exists

This happens when a new socket allocates the same fd as a socket in cleanup process.

Workaround: Set XLIO_TCP_ABORT_ON_CLOSE=1 to lower the chances of such a scenario to take place. 

Keywords: RFS; steering failure error

Discovered in Version: 3.0.2

2970828

Description: XLIO ignores all signals except SIGINT signal, occasionally resulting in blocked IO calls.

Workaround: N/A

Keywords: Signal; SIGINT; IO

Discovered in Version: 2.1.4

3204491

Description: Note the following limitations when working with setsockopt for UDP multicast socket.
1. Setting either IP_DROP_MEMBERSHIP or IP_DROP_SOURCE_MEMBERSHIP reverts IP_MULTICAST_LOOP=0 to 1.
2. Only the first multicast group will be offloaded by XLIO using IP_ADD_SOURCE_MEMBERSHIP or MCAST_JOIN_SOURCE_GROUP. The rest of the groups will be handled by the Kernel.
3. Binding to a specific IP (not any_address) after setting multicast setsockopt options will result in binding to any_address instead.

Workaround: N/A

Keywords: multicast; setsockopt; bind

Discovered in Version: 2.1.4

3217617

Description: When XLIO binds to any IP, resource allocation is required for each IP per port. This action may take up to several seconds with a limit of 64K steerings per family.

Workaround: To avoid this scenario, bind a socket to a specific IP instead. 

Keywords: Bind to any IP; resource allocation

Discovered in Version: 2.0.6

3190147

Description: TCP User Timeout options are not supported with select(); pselect(); poll(); ppoll(); __poll_chk(); and __ppoll_chk.

Workaround: Application may use epoll instead.

Keywords: TCP; user timeout; select; poll; ppoll

Discovered in Version: 2.0.6

3217629

Description: iperf3 usage of longjmp operation in the signal handler results in abandoning previous thread context, leaving the state of internal objects in an undefined state.

Workaround: N/A

Keywords: iperf3; signal handler termination

Discovered in Version: 2.0.6

3231754/3208316

Description: Traffic over IPv6 link-local addresses is not supported.

Workaround: N/A

Keywords: IPv6; link-local

Discovered in Version: 2.0.6

3231768

Description: IPv6 Flow Labels are not supported.

Workaround: N/A

Keywords: IPv6 flow labels

Discovered in Version: 2.0.6

3231781

Description: The following IPv6 multicast socket options are not supported:

• IPV6_MULTICAST_LOOP

• MCAST_BLOCK_SOURCE

• MCAST_UNBLOCK_SOURCE

Workaround: N/A

Keywords: IPV6_MULTICAST_LOOP; MCAST_BLOCK_SOURCE; MCAST_UNBLOCK_SOURCE; socket; IPv6

Discovered in Version: 2.0.6

3232106

Description: SO_MAX_PACING_RATE socket option is not supported.

Workaround: N/A

Keywords: SO_MAX_PACING_RATE; socket

Discovered in Version: 2.0.6

3136901

Description: If a multi-threaded application calls exit() syscall while other threads call socket API concurrently, this can lead to a race between XLIO library destructor and intercepted socket API or in rare cases to a segmentation fault.

Workaround: It is recommended to finish running threads before process termination. Alternatively, avoid calling socket API concurrently with exit() syscall.

Keywords: multithreaded applications

Discovered in Version: 1.3.5

2667588

Description: XLIO conf file does not support IPv6 (see

).

Workaround: N/A

Keywords: xlio.conf; IPv6

Discovered in Version: 1.2.9

N/A

Description: Socket API breaks for closed UDP sockets when using XLIO_NGINX_UDP_POOL_SIZE != 0, as the socket is not entirely closed. Socket operations will not fail after the socket is closed.

Workaround: N/A

Keywords: UDP; pool

Discovered in Version: 1.2.9

1542628

Description: Device memory programming is not supported on VMs that lack Blue Flame support.

Workaround: N/A

Keywords: MEMIC, Device Memory, Virtual Machine, Blue flame

Discovered in Version: 1.0.6

—

Description: If XLIO runs when XLIO_HANDLE_SIGINTR is enabled, an error message might be written upon exiting.

Workaround: Ignore the error message, or run XLIO with XLIO_HANDLE_SIGINTR disabled.

Keywords: XLIO_HANDLE_SIGINTR, error message

Discovered in Version: 1.0.6

965237

Description: The following sockets APIs are directed to the OS and are not offloaded by XLIO:

·        int socketpair(int domain, int type, int protocol, int sv[2]);

·        int dup(int oldfd);

·        int dup2(int oldfd, int newfd);

Workaround: N/A

Keywords: sockets, socketpair, dup, dup2

965227

Description: Multicast (MC) loopback within a process is not supported by XLIO:

• If an application process opens 2 (or more) sockets on the same MC group they will not get each other's traffic.
  Note: MC loopback between different XLIO processes always work.

• Both sockets will receive all ingress traffic coming from the wire

Workaround: N/A

Keywords: Multicast, Loopback

965227

Description: MC loopback between XLIO and the OS limitation.

• The OS will reject loopback traffic coming from the NIC

• MC traffic from the OS to XLIO is functional

Workaround: N/A

Keywords: Multicast, Loopback

965227

Description: MC loopback Tx is currently disabled and setsockopt (IP_MULTICAST_LOOP) is not supported.

Workaround: N/A

Keywords: Multicast, Loopback

1011005

Description: XLIO SELECT option supports up to 200 sockets in TCP.

Workaround: Use ePoll that supports up to 6000 sockets.

Keywords: SockPerf

977899

Description: An unsuccessful trial to connect to a local interface, is reported by XLIO as Connection timeout rather than Connection refused.

Workaround: N/A

Keywords: Verification

1019085

Description: Poll is limited with the number of sockets.

Workaround: Use ePoll for large number of sockets (tested up to 6000)

Keywords: Poll, ePoll

—

Description: The following XLIO_PANIC will be displayed when there are not enough open files defined on the server:

XLIO PANIC  : si[fd=1023]:51:sockinfo() failed to create internal epoll (ret=-1 Too many open files

Workaround: Verify that the number of max open FDs (File Descriptors) in the system (ulimit -n) is twice as number of needed sockets. XLIO internal logic requires one additional FD per offloaded socket.

Keywords: XLIO_PANIC while opening large number of sockets

—

Description: When a XLIO -enabled application is running, there are several cases when it does not exit as expected pressing CTRL-C.

Workaround: Enable SIGINT handling in XLIO, by using:

#export XLIO_HANDLE_SIGINTR=1

Keywords: The XLIO application does not exit when you press CTRL-C.

—

Description: XLIO does not support network interface changes during runtime

Workaround: N/A

Keywords: Dynamic network interface changes

—

Description: XLIO behavior of epoll EPOLLET (Edge Triggered) and EPOLLOUT flags with TCP sockets differs between OS and  XLIO.

• XLIO – triggers EPOLLOUT event every received ACK (only data, not syn/fin)

• OS – triggers EPOLLOUT event only after buffer was full

Workaround: N/A

Keywords: XLIO behavior of epoll EPOLLET (Edge Triggered) and EPOLLOUT flags with TCP sockets

—

Description: XLIO does not close connections located on the same node (sends FIN to peers) when its own process is terminated.

Workaround: N/A

Keywords: XLIO connection

—

Description: XLIO is not closed (sends FIN to peers) when its own process is terminated when the /etc/init.d/XLIO is stopped.

Workaround: Launch /etc/init.d/XLIO start

Keywords: XLIO connection

—

Description: When a non-offloaded process joins the same MC address as another XLIO process on the same machine, the non-offloaded process does not get any traffic.

Workaround: Run both processes with  XLIO

Keywords: MC traffic with XLIO process and non XLIO process on the same machine

—

Description: Occasionally, epoll with EPOLLONESHOT does not function properly. 

Workaround: N/A

Keywords: epoll with EPOLLONESHOT

—

Description: Occasionally, when running UDP SFNT-STREAM client with poll muxer flag, the client-side ends with an expected error:

ERROR: Sync messages at end of test lost
ERROR: Test failed.

This only occurs with poll flag

Workaround: Set a higher acknowledgment waiting time value in the sfnt-stream.

Keywords: SFNT-STREAM UDP with poll muxer flag ends with an error on client-side

—

Description: Occasionally, SFNT-STREAM UDP client hangs when running multiple times.

Workaround: Set a higher acknowledgment waiting time value in the sfnt-stream.

Keywords: SFNT-STREAM UDP client hanging issue

—

Description: Ethernet loopback functions only if both sides are either off-loaded or not-offloaded.

Workaround: N/A

Keywords: Ethernet loopback is not functional between the XLIO and the OS

—

Description: The following error may occur when running netperf TCP tests with XLIO:

remote error 107 'Transport endpoint is not connected

Workaround: Use netperf 2.6.0

Keywords: Error when running netperf 2.4.4 with  XLIO

—

Description: Occasionally, a packet is not sent if the socket is closed immediately after send() (also for blocking socket)

Workaround: Wait several seconds after send() before closing the socket

Keywords: A packet is not sent if the socket is closed immediately after send()

—

Description: It can take for XLIO more time than the OS to return from an iomux call if all sockets in this iomux are empty sockets

Workaround: N/A

Keywords: Iomux call with empty sockets

—

Description: Sharing of HW resources between the different working threads might cause lock contentions which can affect performance.

Workaround: Use different ring allocation logic.

Keywords: Issues with performance with some multi-threaded applications

—

Description: XLIO does not support broadcast traffic.

Workaround: Use libxlio.conf to pass broadcast through OS

Keywords: No support for direct broadcast

—

Description: Directing XLIO to access non-valid memory area will cause a segmentation fault.

Workaround: N/A

Keywords: There is no non-valid pointer handling in  XLIO

—

Description: XLIO allocates resources on the first connect/send operation, which might take up to several tens of milliseconds.

Workaround: N/A

Keywords: First connect/send operation might take more time than expected

—

Description: Calling select upon shutdown of socket will return “ready to write” instead of timeout.

Workaround: N/A

Keywords: Calling select() after shutdown (write) returns socket ready to write, while select() is expected to return timeout

—

Description: XLIO does not raise sigpipe in connection shutdown.

Workaround: N/A

Keywords: XLIO does not raise sigpipe

—

Description: XLIO polls the CQ for packets; if no packets are available in the socket layer, it takes longer.

Workaround: N/A

Keywords: When there are no packets in the socket, it takes longer to return from the read call

—

Description: Select with more than 1024 sockets is not supported

Workaround: Compile XLIO with SELECT_BIG_SETSIZE defined.

Keywords: 1024 sockets

—

Description: Possible performance degradation running NGINX QUIC.

Workaround: Use XLIO_CQ_MODERATION_ENABLE=0.

Keywords: QUIC, NGINX, XLIO_CQ_MODERATION_ENABLE