Networking NVIDIA Accelerated IO (XLIO) Documentation
Networking NVIDIA Accelerated IO (XLIO) Documentation
Breadcrumbs

Appendix: Nginx

Table of Contents

  • Introduction

  • NGINX Build Instructions

  • NGINX Configuration

  • TLS Setup

  • System Configuration

    • NUMA Considerations

    • Huge Pages Configuration

  • Running NGINX over XLIO

  • Troubleshooting

    • Client-Side Checks

Introduction

This guide covers deploying NGINX over NVIDIA Accelerated IO (XLIO), focusing on best performance practices and recommended conventions.

Nginx is a web server that is simple to configure for serving static web content and can also be deployed to deliver dynamic content across networks. For more information, visit the official Nginx website.

NGINX & OPENSSL Build Instructions

We recommend using stable, official versions of NGINX and OpenSSL.

# mkdir /opt/nginx_xlio
# cd /opt/nginx_xlio
# git clone https://github.com/openssl/openssl.git -b openssl-3.0.2 
# git clone https://github.com/nginx/nginx.git -b release-1.21.6
# cd nginx
# auto/configure --prefix=/opt/nginx_xlio/install --with-openssl=/opt/nginx_xlio/openssl --with-http_ssl_module --with-http_v2_module --with-openssl-opt="enable-ktls -DOPENSSL_LINUX_TLS"
# make -j && make install

Setup Build Instructions

This example can be adapted to your specific requirements.

worker_processes 16; # this directive needs to be coherent with XLIO_NGINX_WORKERS_NUM
daemon off;
user root root;
worker_rlimit_nofile 1048575;
worker_priority -20;
error_log /dev/stdout info;
pid logs/nginx.pid;

events {
    worker_connections 200000;
    use epoll;
    multi_accept off;
    accept_mutex off;
}

http {
    include mime.types;
    default_type application/octet-stream;
    access_log off;
    client_body_timeout 1800s;
    client_header_timeout 1800s;
    send_timeout 1800s;
    keepalive_timeout 1h;
    keepalive_requests 100000000;

	# Adjust to XLIO logic
    ssl_buffer_size 16128; 
    
	# If using “sendfile on”, add the following directives to the http block:
	sendfile on;
	open_file_cache max=1000 inactive=20s;
	open_file_cache_valid 3600s;
	open_file_cache_min_uses 2;
	open_file_cache_errors on;

    server {
        listen [SPECIFIC_IPV4_ADDR]:443 ssl default_server backlog=65535;
        listen [SPECIFIC_IPV6_ADDR]:443 ssl default_server backlog=65535;
        server_name localhost;
        ssl_certificate /etc/ssl/certs/nginx-rsa-selfsigned.crt;
        ssl_certificate_key /etc/ssl/private/nginx-rsa-selfsigned.key;
        ssl_conf_command Options KTLS; # to enable KTLS usage with NGINX - comment the directive to disable KTLS.
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES256-GCM-SHA384:AES128-GCM-SHA256";
        ssl_conf_command Ciphersuites "TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256";
        ssl_prefer_server_ciphers on;
        location / {
            root html;
            index index.html index.htm;
        }
    }
}

Note: XLIO doesn't support daemon on. Keep it off.

TLS Setup

To utilize TLS HW offload – please check for TLS HW offload requirements and supported Ciphers please see  NVIDIA Accelerated IO (XLIO) Documentation Rev 3.40.2 - NVIDIA Docs → “Advanced Features” →” TLS HW offload”

System Configuration

Numa Considerations

Aligning your application with the NUMA node of the NVIDIA card reduces cross-node memory access latency, enhancing performance.

Steps to Check NUMA Node for a Network Interface:

  • Check NUMA Node of Interface:

#<host> ip addr show
#<host> cat /sys/class/net/<interface_name>/device/numa_node

  • Bind Application to NUMA Node:

#<host> numactl --cpunodebind=<NUMA_NODE> <your_application>


Huge Pages Configuration

XLIO can leverage Huge Pages to reduce TLB misses and improve memory allocation efficiency.

  • Check Supported Huge Page Sizes:

#(host) ls /sys/kernel/mm/hugepages/

  • Allocate Huge Pages (approx. 2GB per NGINX worker is Recommended):

#(host) echo <number_of_hugepages> | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
OR
#(host) echo <number_of_hugepages> | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

  • NUMA-aware Huge Pages Allocation (Recommended):

we recommend that Huge Pages are allocated on the same NUMA node as the application, memory access latency is reduced, and overall performance is optimized.

#(host) echo <number_of_hugepages> | sudo tee /sys/devices/system/node/<NUMA_NODE>/hugepages/hugepages-2048kB/nr_hugepages
OR
#(host) echo <number_of_hugepages> | sudo tee /sys/devices/system/node/<NUMA_NODE>/hugepages/hugepages-1048576kB/nr_hugepages

Running NGINX over XLIO 

#<host> ulimit -l unlimited

Example XLIO configuration for X86

#<host> export XLIO_SPEC=nginx
#<host> export XLIO_NGINX_WORKERS_NUM=16
#<host> export XLIO_TX_BUF_SIZE=16384
#<host> LD_PRELOAD=/path/to/libxlio.so /path/to/nginx -c /path/to/nginx.conf



#<host> XLIO_SPEC=nginx XLIO_NGINX_WORKERS_NUM=16 <MORE_XLIO_PARAMS>  LD_PRELOAD=path/to/libxlio.so path/to/nginx -c path/to/nginx.conf

Example XLIO configuration for aarch64 (Bluefield)

#<host> export XLIO_SPEC=nginx_dpu
#<host> export XLIO_NGINX_WORKERS_NUM=16
#<host> export XLIO_TX_BUF_SIZE=16384
#<host> LD_PRELOAD=/path/to/libxlio.so /path/to/nginx -c /path/to/nginx.conf


Note: -

  • XLIO has many other optimization options that can be adjusted to maximize performance for specific use cases.

  • to monitoring XLIO performance counters, please see Documentation → ”Monitoring,Debugging,and Troubleshooting”


Troubleshooting

Client-Side Checks

Ensure client:

    • Is not CPU- or memory-bound

    • Can receive at full rate without dropping packets

    • Doesn’t trigger excessive retransmissions

Last updated: