Networking NVIDIA Accelerated IO (XLIO) Documentation
Networking NVIDIA Accelerated IO (XLIO) Documentation
Breadcrumbs

Configure XLIO Inside Docker Container

Table of Contents

Introduction

  • This guide offers instructions for running NVIDIA Accelerated IO (XLIO) in Docker containers using NVIDIA DOCA Software Framework.

  • NVIDIA NICs utilize Single Root IO Virtualization (SR-IOV), a technology that allows a physical NIC (PF) to present multiple virtual instances (Virtual Functions or VFs).

    • Each VF is a lightweight instance of the PF appearing as a distinct network interface with no additional overhead.

    • PFs are typically used in the host network stack, while VFs are typically used in virtualized environments like Virtual Machines and Docker containers.

Prerequisites

  • Check XLIO System Requirements for details on supported NICs and more.

  • Ensure all necessary kernel drivers for NVIDIA hardware, including Mellanox NICs, are installed on the host before deploying containers.

  • If SR-IOV is required, ensure it is enabled on SR-IOV-capable NIC on the host. Refer Docker Using SR-IOV for more instructions.

XLIO Library Installation Using DOCA_HOST

Ensure you use the latest DOCA_HOST version.

Install libxlio Using DOCA HOST Inside the Container

  1. Pull the Desired Container Image

    #(host) docker pull <container-image> # e.g. docker pull ubuntu:22.04

  2. Run the Container Image

    #(host) docker run -it <container-image> /bin/bash

  3. Install Required Packages Inside the Container

    • For DEB-Based Distributions (e.g. Ubuntu):#(container) <your-package-manager> update #(container) <your-package-manager> -y install curl gpg iproute2 iputils-ping

    • For RPM-Based Distributions (e.g. RH and Oracle Linux):#(container) <your-package-manager> update #(container) <your-package-manager> -y install iproute iputils

  4. Install libxlio Library and Its Dependencies Inside the Container

    • Visit NVIDIA DOCA Downloads | NVIDIA Developer page.

    • Select the deployment package - DOCA-HOST.

    • Select the profile - doca-all.

    • Select your container distribution.

    • Select installer type - (Online).

    • Follow the installation instructions with a minor change in the last instruction: instead of sudo <your-package-manager> -y install doca-all, use:#(container) <your-package-manager> -y install libxlio

  5. Install sockperf Inside the Container

    #(container) <your-package-manager> -y install sockperf

  6. Exit and Save the Container Image

    • Exit the container:#(container) exit

    • Identify the container ID and commit the container to a new image:#(host) docker ps -a #(host) docker commit -m "Added XLIO" <CONTAINER-ID> xlio-image

Container Setup

This section provides instructions on how to run and configure the newly created xlio-image Docker image, detailing-

both the required and some of the optional configurations for running the container.

Required Configurations

  • --ulimit memlock=-1: This option allows you to set unlimited memory lock for the container. For more details, refer to the ULIMIT Considerations section.

  • --device=/dev/infiniband: This option grants the container access to available InfiniBand devices.

    Note: Instead of granting access to all available IB devices, you can grant access to specific InfiniBand devices by using:

    --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0

    (Make sure to use the correct uverbsX in case you have multiple InfiniBand devices).

  • --cap-add=NET_RAW NET_ADMIN: This capabilities provides the ability to configure and manage network interfaces, in addition to Raw Socket processing.

Optional Configurations

XLIO Hugepages Configuration

XLIO can take advantage of Huge Pages to optimize memory allocation and takes full advantage of the performance benefits of huge pages.

Huge Pages are allocated by the host. Once configured, the container can access and use these Huge Pages from the host's memory pool.

  1. Check Current Huge Page Configuration

    #(host) cat /proc/meminfo | grep HugePages

  2. Allocate Sufficient Amount of Huge Pages (Preferably Total of 2GB Memory)

    To allocate a total of approximately 2GB of huge pages, determine the size of your
    system's hugepages (usually 2MB) and calculate the required number.
    For example, if your system uses 2MB huge pages, you would need to allocate 1024 huge pages to reach a total of 2GB.

    #(host) echo <number_of_hugepages> | sudo tee /proc/sys/vm/nr_hugepages

Network Configurations

Network Configuration Option 1: Host Network

Use the host's network stack, which directly connects the container to the host's networking environment:

  1. Run The Container (xlio-image)

    #(host) docker run -it --net=host --cap-add=NET_RAW --cap-add=NET_ADMIN --ulimit memlock=-1 --device=/dev/infiniband xlio-image /bin/bash


Network Configuration Option 2: Custom SR-IOV Docker Network

To run the container in a separate network namespace from the host, you can use a custom Docker network.
NVIDIA provides an SR-IOV Docker plugin that facilitates the creation and management of such networks by
automatically allocating & assigning a Virtual Function (VF) to the container.

  • Please ensure that the requirements outlined in the Prerequisites section regarding SR-IOV enablement have been met.

Limitations:

    1. NVIDIA Docker SR-IOV Plugin is supported ONLY on Linux environment on x86_64 and ppc64le platforms.

    2. Using a separate network namespace limits access to some /proc/sys net.core parameters fetched by XLIO, causing
      it to fall back on hardcoded default values and push a warning; a solution is in development.

    3. SR-IOV Plugin ONLY works with ConnectX series And Bluefield in NIC mode.

  1. QuickStart Instructions for Creating a Custom Docker Network with SR-IOV plugin

a. Ensure you are using Docker 1.9 or later.

b. Pull the SR-IOV Plugin (Mellanox/docker-sriov-plugin)

#(host) docker pull rdma/sriov-plugin

c. Run the Plugin.

#(host) docker run -v /run/docker/plugins:/run/docker/plugins -v /etc/docker:/etc/docker -v /var/run:/var/run --net=host --privileged rdma/sriov-plugin

d. Create a new Docker Network using the SR-IOV plugin as driver.

For example, using the ens2f0 PF-based Net device (Must be a PF-based interface):

#(host) docker network create --driver sriov --subnet=<subnet> --gateway=<default-gateway-ip> -o netdevice=ens2f0 -o privileged=1 mynet

Notes:

      • If the custom network subnet has a default gateway, use it as <default-gateway-ip>; it will provide external connectivity. Otherwise, ignore the --gateway option.

      • It is important to create the SR-IOV Docker network after the SR-IOV plugin is already running.

  2. Run The Container (xlio-image)

#(host) docker run -it --net=mynet --ip=<picked-VF-interface-IP> --cap-add=NET_RAW --cap-add=NET_ADMIN --ulimit memlock=-1 --device=/dev/infiniband xlio-image /bin/bash

Note: Ensure that the IP address you assign with --ip is a free IP address in the subnet to avoid conflicts within the subnet.

       3. Verify Successfully Assigned VF network interface inside the container 

#(container) ip addr show 

Optional: Connect to additional Docker network for external access

If the custom SR-IOV network does not have a default gateway and if you need access to external networks, you can

Connect the container to an additional Docker network (e.g. second SR-IOV network or default bridge network) to provide external access:

    1. Identify the Container ID and Connect xlio-image Container to the Bridge Network

      #(host) docker ps -a #(host) docker network connect bridge <container-id>

    2. Verify That the Container Gained a New Network Interface to the Bridge Network

      #(container) ip addr show

    3. Verify the Default Route Uses the Bridge Network

      #(container) ip route

      • If the default gateway is incorrect, update the default route inside the container:

        1. Fetch Bridge Network Gateway IP

          #(host) docker network inspect bridge | grep Gateway

        2. Update the Default Route

          #(container) ip route del default #(container) ip route add default via <bridge-gateway-ip> dev <bridge-interface>

Verification of XLIO Functionality

Verify that XLIO is working within the container:

  1. Run Sockperf Server on a Different Server

    # LD_PRELOAD=libxlio.so sockperf server -i <SERVER_IP>

  2. Run the Sockperf Client with XLIO Inside the Container

    Using the xlio-image container that you started in the setup section, run the sockperf client with XLIO Library to test connectivity and performance:

    #(container) LD_PRELOAD=libxlio.so sockperf pp -i <SERVER_IP> -t 5

    • <SERVER_IP> should be replaced with the IP address of the server running the sockperf server.

    Note: This is not a fully optimized XLIO run, just a sanity check. XLIO has many optimization options, but they are outside the scope of this section.

ULIMIT Considerations

XLIO requires a much higher max locked memory ulimit (ulimit -l) than the default. A container does not

inherit the ulimits from the host (unless running in privileged mode), and changing the ulimit value within the

container is not allowed.Therefore, it is preferable to set it to unlimited by running the container with:

--ulimit memlock=-1

where -1 means unlimited memory lock.

Another option is to set the default ulimit value for the Docker daemon, which containers
inherit (running a container with --ulimit will override the daemon --default_ulimit).

Last updated: