SNAP-4 Service Appendixes

1. Appendix – DPU Firmware Configuration

Before configuring SNAP, the user must ensure that all firmware configuration requirements are met. By default, SNAP is disabled and must be enabled by running both common SNAP configurations and additional protocol-specific configurations depending on the expected usage of the application (e.g., hot-plug, SR-IOV, UEFI boot, etc).

After configuration is finished, the host must be power cycled for the changes to take effect. 

mlxconfig -d /dev/mst/mt41692_pciconf0 -e query

1.1. System Configuration Parameters

1.2. RDMA/RoCE Configuration

By default, BlueField's RDMA/RoCE communication is blocked for its primary OS interfaces (known as ECPFs, typically mlx5_0 and mlx5_1).

If RoCE traffic is required, you must create additional network functions (scalable functions) that support RDMA/RoCE.

To enable RoCE interfaces, run the following from within the DPU:

[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PER_PF_NUM_SF=1
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2
[dpu] mlxconfig -d /dev/mst/mt41692_pciconf0.1 s PF_SF_BAR_SIZE=8 PF_TOTAL_SF=2

1.3. NVMe Configuration

1.4. Virtio-blk Configuration

2. Appendix – Configure Persistent Network Interfaces

To configure persistent network interfaces so they are not lost after reboot. Under /etc/sysconfig/network-scripts modify the following four files, or create them if do not exist, then perform a reboot:

# cd /etc/sysconfig/network-scripts/
# cat ifcfg-p0
  NAME="p0"
  DEVICE="p0"
  NM_CONTROLLED="no"
  DEVTIMEOUT=30
  PEERDNS="no"
  ONBOOT="yes"
  BOOTPROTO="none"
  TYPE=Ethernet
  MTU=9000

# cat ifcfg-p1
  NAME="p1"
  DEVICE="p1"
  NM_CONTROLLED="no"
  DEVTIMEOUT=30
  PEERDNS="no"
  ONBOOT="yes"
  BOOTPROTO="none"
  TYPE=Ethernet
  MTU=9000

# cat ifcfg-enp3s0f0s0
  NAME="enp3s0f0s0"
  DEVICE="enp3s0f0s0"
  NM_CONTROLLED="no"
  DEVTIMEOUT=30
  PEERDNS="no"
  ONBOOT="yes"
  BOOTPROTO="static"
  TYPE=Ethernet
  IPADDR=1.1.1.1
  PREFIX=24
  MTU=9000

# cat ifcfg-enp3s0f1s0
  NAME="enp3s0f1s0"
  DEVICE="enp3s0f1s0"
  NM_CONTROLLED="no"
  DEVTIMEOUT=30
  PEERDNS="no"
  ONBOOT="yes"
  BOOTPROTO="static"
  TYPE=Ethernet
  IPADDR=1.1.1.2
  PREFIX=24
  MTU=9000

3. Appendix – Building SNAP Container with Custom SPDK

The SNAP source package contains the files necessary for building a container with a custom SPDK. 

To build the container:

1. Download and install the SNAP sources package:

   [dpu] # dpkg -i /path/snap-sources_<version>_arm64.deb
   

2. Navigate to the src folder and use it as the development environment:

   [dpu] # cd /opt/nvidia/nvda_snap/src
   

3. Copy the following to the container folder:

   • SNAP source package – required for installing SNAP inside the container

   • Custom SPDK – to container/spdk. For example: 

     [dpu] # cp /path/snap-sources_<version>_arm64.deb  container/
     [dpu] # git clone -b v23.01.1 --single-branch --depth 1 --recursive --shallow-submodules https://github.com/spdk/spdk.git container/spdk
     

4. Modify the spdk.sh file if necessary as it is used to compile SDPK.

5. To build the container:

   • For Ubuntu, run:

     [dpu] # ./container/build_public.sh --snap-pkg-file=snap-sources_<version>_arm64.deb
     

   • For CentOS, run:

     [dpu] # rpm -i snap-sources-<version>.el8.aarch64.rpm
     [dpu] # cd /opt/nvidia/nvda_snap/src/
     [dpu] # cp /path/snap-sources_<version>_arm64.deb container/
     [dpu] # git clone -b v23.01.1 --single-branch --depth 1 --recursive --shallow-submodules https://github.com/spdk/spdk.git container/spdk
     [dpu] # yum install docker-ce docker-ce-cli
     [dpu] # ./container/build_public.sh --snap-pkg-file=snap-sources_<version>_arm64.deb
     

6. Transfer the created image from the Docker tool to the crictl tool. Run:

   [dpu] # docker save doca_snap:<version> doca_snap.tar
   [dpu] # ctr -n=k8s.io images import doca_snap.tar
   

   To transfer the container image to other setups, refer to appendix "SNAP-4 Service Appendixes | id (3.4.0)SNAP 4ServiceAppendixes DeployingContaineronSetupsWithoutInternetConnectivity".

7. To verify the image, run:

   [DPU] #  crictl images
   IMAGE                                  TAG                   IMAGE ID            SIZE
   docker.io/library/doca_snap            <version>             79c503f0a2bd7       284MB
   

8. Edit the image filed in the container/doca_snap.yaml file. Run:

   image: doca_snap:<version>
   

9. Use the YAML file to deploy the container. Run:

   [dpu] # cp doca_snap.yaml /etc/kubelet.d/
   

   The container deployment preparation steps are required.

4. Appendix – Deploying Container on Setups Without Internet Connectivity

When Internet connectivity is not available on a DPU, Kubelet scans for the container image locally upon detecting the SNAP YAML. Users can load the container image manually before the deployment.

To accomplish this, users must download the necessary resources using a DPU with Internet connectivity and subsequently transfer and load them onto DPUs that lack Internet connectivity.

1. To download the .yaml file: 

   [bf] # wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/doca/doca_container_configs/versions/<path-to-yaml>/doca_snap.yaml
   

   Access the latest download command on NGC. The doca_snap:4.1.0-doca2.0.2 tag is used in this section as an example, and the latest tag is also available on NGC.

2. To download SNAP container image:

   [bf] # crictl pull nvcr.io/nvidia/doca/doca_snap:4.1.0-doca2.0.2
   

3. To verify that the SNAP container image exists:

   [bf] # crictl images
   
   IMAGE                                  TAG                 IMAGE ID            SIZE
   nvcr.io/nvidia/doca/doca_snap          4.1.0-doca2.0.2     9d941b5994057       267MB
   k8s.gcr.io/pause                       3.2                 2a060e2e7101d       251kB
   

   k8s.gcr.io/pause image is required for the SNAP container.

4. To save the images as a .tar file:

   [bf] # mkdir images
   [bf] # ctr -n=k8s.io image export images/snap_container_image.tar nvcr.io/nvidia/doca/doca_snap:4.1.0-doca2.0.2
   [bf] # ctr -n=k8s.io image export images/pause_image.tar k8s.gcr.io/pause:3.2
   

5. Transfer the .tar files and run the following to load them into Kubelet:

   [bf] # sudo ctr --namespace k8s.io image import images/snap_container_image.tar
   [bf] # sudo ctr --namespace k8s.io image import images/pause_image.tar
   

6. Now, the image exists in the tool and is ready for deployment.

   [bf] # crictl images
   IMAGE                                  TAG                 IMAGE ID            SIZE
   nvcr.io/nvidia/doca/doca_snap          4.1.0-doca2.0.2     9d941b5994057       267MB
   k8s.gcr.io/pause                       3.2                 2a060e2e7101d       251kB 
   

5. Appendix – Install Legacy SPDK

To build SPDK-19.04 for SNAP integration:

1. Cherry-pick a critical fix for SPDK shared libraries installation (originally applied on upstream only since v19.07). 

   [spdk.git] git cherry-pick cb0c0509
   

2. Configure SPDK:

   [spdk.git] git submodule update --init
   [spdk.git] ./configure --prefix=/opt/mellanox/spdk --disable-tests --without-crypto --without-fio --with-vhost --without-pmdk --without-rbd --with-rdma --with-shared --with-iscsi-initiator --without-vtune
   [spdk.git] sed -i -e 's/CONFIG_RTE_BUILD_SHARED_LIB=n/CONFIG_RTE_BUILD_SHARED_LIB=y/g' dpdk/build/.config
   

   The flags --prefix, --with-rdma, and --with-shared are mandatory.

3. Make SPDK (and DPDK libraries):

   [spdk.git] make && make install
   [spdk.git] cp dpdk/build/lib/* /opt/mellanox/spdk/lib/
   [spdk.git] cp dpdk/build/include/* /opt/mellanox/spdk/include/
   

6. Appendix – PCIe BDF to VUID Translation

PCIe BDF (Bus, Device, Function) is a unique identifier assigned to every PCIe device connected to a computer. By identifying each device with a unique BDF number, the computer's OS can manage the system's resources efficiently and effectively.

PCIe BDF values are determined by host OS and are hence subject to change between different runs, or even in a single run. Therefore, the BDF identifier is not the best fit for permanent configuration.

To overcome this problem, NVIDIA devices add an extension to PCIe attributes, called VUIDs. As opposed to BDF, VUID is persistent across runs which makes it useful as a PCIe function identifier.

PCI BDF and VUID can be extracted one out of the other, using lspci command:

1. To extract VUID out of BDF: 

   [host] lspci -s <BDF> -vvv | grep -i VU | awk '{print $4}'
   

2. To extract BDF out of VUID: 

   [host] ./get_bdf.py <VUID>
   [host] cat ./get_bdf.py
   #!/usr/bin/python3 
    
   import subprocess
   import sys
    
   vuid = sys.argv[1]
    
   # Split the output into individual PCI function entries
   lspci_output = subprocess.check_output(['lspci']).decode().strip().split('\n')
    
   # Create an empty dictionary to store the results
   pci_functions = {}
    
   # Loop through each PCI function and extract the BDF and full info
   for line in lspci_output:
       bdf = line.split()[0]
       if vuid in subprocess.check_output(['lspci', '-s', bdf, '-vvv']).decode():
           print(bdf)
           exit(0)
    
   print("Not Found")
   

7. Appendix – SNAP Memory Consumption

This appendix explains how SNAP consumes memory and how to manage memory allocation.

The user must allocate the DPA hugepages memory according to the section "Step 1: Allocate Hugepages". It is possible to use use a portion of the DPU memory allocation in the SNAP container as described in section "Adjusting YAML Configuration". This configuration includes the following minimum and maximum values:

• The minimum allocation which the SNAP container consumes:

  resources:
    requests:
      memory: "4Gi"
  

• The maximum allocation that the SNAP container is allowed to consume:

  resources:
    limits:
      hugepages-2Mi: "4Gi"
  

Hugepage memory is used by the following:

• SPDK mem-size global variable which controls the SPDK hugepages consumption (configurable in SPDK, 1GB by default)

• SNAP SNAP_MEMPOOL_SIZE_MB – used with non-ZC mode as IO buffers staging buffers on the Arm. By default, the SNAP mempool consumes 1G from the SPDK mem-size hugepages allocation. SNAP mempool may be configured using the SNAP_MEMPOOL_SIZE_MB global variable (minimum is 64 MB). 

  If the value assigned is too low, with non-ZC, a performance degradation could be seen.

• SNAP and SPDK internal usage – 1G should be used by default. This may be reduced depending on the overall scale (i.e., VFs/num queues/QD).

• XLIO buffers – allocated only when NVMeTCP XLIO is enabled.

The following is the limit of the container memory allowed to be used by the SNAP container:

resources:
  limits:
    memory: "6Gi"

The SNAP container also consumes DPU SHMEM memory when NVMe recovery is used (described in section "NVMe Recovery"). In addition, the following resources are used:

limits:
  memory:

8. Appendix – Host OS Configuration

With Linux environment on host OS, additional kernel boot parameters may be required to support SNAP related features:

• To use SR-IOV:

  • For Intel, intel_iommu=on iommu=pt must be added

  • For AMD, amd_iommu=on iommu=pt must be added

• To use PCIe hotplug, pci=realloc must be added

• modprobe.blacklist=virtio_blk,virtio_pci for non-built-in virtio-blk driver or virtio-pci driver

To view boot parameter values, run:

cat /proc/cmdline

It is recommended to use the following with virtio-blk:

[dpu] cat /proc/cmdline BOOT_IMAGE … pci=realloc modprobe.blacklist=virtio_blk,virtio_pci

To enable VFs (virtio_blk/NVMe):

echo 125 > /sys/bus/pci/devices/0000\:27\:00.4/sriov_numvfs

8.1. Intel Server Performance Optimizations

cat /proc/cmdline
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet iommu=pt intel_iommu=on pci=realloc modprobe.blacklist=virtio_blk,virtio_pci

8.2. AMD Server Performance Optimizations

cat /proc/cmdline
cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.15.0_mlnx root=UUID=91528e6a-b7d3-4e78-9d2e-9d5ad60e8273 ro crashkernel=auto resume=UUID=06ff0f35-0282-4812-894e-111ae8d76768 rhgb quiet iommu=pt amd_iommu=on pci=realloc modprobe.blacklist=virtio_blk,virtio_pci

9. Appendix – SPDK Configuration

The SPDK bdev_nvme module utilizing the RDMA transport does not support Completion Queue (CQ) resizing.

This constraint limits the number of SPDK NVMe RDMA I/O queues that can be created per core, which consequently restricts the maximum number of supported PCIe functions (including VFs, static PFs, and hot-plug PFs).

The exact limit depends on factors such as the maximum queue depth configured on the NVMe-oF RDMA target and the number of paths to that target. For example, with a maximum queue depth of 128 (the default for SPDK NVMe-oF targets) and a single path per subsystem, the default CQ size supports only up to 16 I/O queues per core.

To support larger-scale deployments, you must either enable a Shared Receive Queue (SRQ) or configure a larger CQ size.

9.1. Enable SRQ

The bdev_nvme module can be configured to use a SRQ. This allows multiple connections to share receive buffers, significantly increasing scalability. This is the recommended approach, though it may result in a minor IOPS reduction (around 5%) at exceptionally large scales (more than 500 queue pairs per core).

To enable SRQ, set the --rdma-srq-size option via the bdev_nvme_set_options RPC:

spdk_rpc.py bdev_nvme_set_options --rdma-srq-size <RDMA_SRQ_SIZE>

The SRQ size must be large enough to minimize Receiver Not Ready (RNR) events, which degrade performance. The optimal value depends on your specific traffic profile.

9.2. Configure CQ Size

The bdev_nvme_set_options RPC provides two parameters to set the initial and maximum CQ size: --rdma-initial-cq-size and --rdma-max-cq-size. 

The CQ size must be large enough to accommodate all Completion Queue Entries (CQEs) across all queue pairs. You can estimate the maximum number of outstanding CQEs using the formula .

Usage example:

spdk_rpc.py bdev_nvme_set_options --rdma-initial-cq-size <RDMA_CQ_SIZE> --rdma-max-cq-size <RDMA_CQ_SIZE>

9.3. Disable bdev Read/Write Bypass for Large VF Scale

In large-scale deployments (e.g., 250+ Virtual Functions), SNAP and SPDK configurations utilizing virtio-blk or NVMe backends may experience significant read throughput degradation if the SPDK bdev read/write bypass path is enabled. Disabling this bypass restores read performance in these scenarios. Apply this configuration when tuning large VF deployments for read-heavy workloads. 

To disable the bypass path, run:

spdk_rpc.py bdev_set_options --disable-rw-bypass

To re-enable the bypass path (which is recommended for all standard, non-scaled configurations), run:

spdk_rpc.py bdev_set_options --enable-rw-bypass

10. Appendix – Custom Bdev Modules

This section explains how to extend the SNAP service with custom SPDK Bdev modules. The guide provides two distinct workflows:

1. Production Build: Creating a new container image that permanently includes the custom module.

2. Development Build: Manually building and running inside a container for rapid iteration and testing.

10.1. Method 1: Building a Production Container (Recommended)

This method generates a standalone container image with the SNAP service and your custom functionality built-in, making it suitable for deployment via Kubernetes.

10.1.1. Environment Preparation

Create a working directory and clone the SPDK source code corresponding to your SNAP version.

# Create working directory (example: /opt/build)
mkdir -p /opt/build
cd /opt/build

# Clone SPDK sources
git clone https://github.com/Mellanox/spdk --branch v25.05.2.nvda

# Verify directory structure
ls -F
# Output should show: Dockerfile  spdk/

10.1.2. Create the Dockerfile

Create a Dockerfile in your working directory with the following content. This multi-stage build compiles the module and then copies it into the final SNAP image.

# Stage 1: Builder
FROM nvcr.io/nvstaging/doca/doca_snap:4.9.0-6-doca3.2.0 as builder

# Install build dependencies
RUN apt-get update && apt-get install -y \
    autoconf libtool python3-pyelftools libaio-dev \
    libncurses-dev libfuse3-dev patchelf libcmocka-dev make

# Copy external code source
COPY spdk/test/external_code /external_code

# Build the custom module
WORKDIR /external_code
ENV SPDK_HEADER_DIR=/opt/mellanox/spdk/include
ENV SPDK_LIB_DIR=/opt/mellanox/spdk/lib
ENV DPDK_LIB_DIR=/opt/mellanox/spdk/include

RUN make passthru_shared

# Stage 2: Final Image
FROM nvcr.io/nvstaging/doca/doca_snap:4.9.0-6-doca3.2.0

# Copy compiled shared object
COPY --from=builder /external_code/passthru/libpassthru_external.so /opt/nvidia/spdk/lib/
ENV SNAP_LD_PRELOAD=/opt/nvidia/spdk/lib/libpassthru_external.so

# Copy RPC python script
COPY --from=builder /external_code/passthru/bdev_passthru.py /usr/lib/python3/dist-packages/spdk/rpc/
ENV SPDK_RPC_PLUGIN="spdk.rpc.bdev_passthru"

10.1.3. Build and Deploy

1. Build the container image:

   Bash

   docker build -t doca_snap_custom_bdev:latest -f Dockerfile .
   

2. Import to Kubernetes (Kubelet):

   Bash

   # Export image to tarball
   docker save doca_snap_custom_bdev:latest > doca_snap_custom_bdev.tar
   
   # Import into containerd
   ctr -n=k8s.io images import doca_snap_custom_bdev.tar
   

3. Edit /etc/kubelet.d/doca_snap.yaml to use the new image:

   image: doca_snap_custom_bdev:latest
   

10.1.4. Configuration Example

Once the container is running, you can configure the custom bdev module via RPC. 

# Identify the running SNAP container ID
SNAP_ID=$(crictl ps -s running -q --name snap)

# Create a base Malloc bdev
crictl exec $SNAP_ID spdk_rpc.py bdev_malloc_create 32 4096 -b Malloc0

# Construct the custom passthru bdev on top of Malloc0
crictl exec $SNAP_ID spdk_rpc.py construct_ext_passthru_bdev -b Malloc0 -p TestPT

# Create NVMe Subsystem and Controller
crictl exec $SNAP_ID snap_rpc.py nvme_subsystem_create --nqn nqn.2023-05.io.nvda.nvme:0
crictl exec $SNAP_ID snap_rpc.py nvme_controller_create --nqn nqn.2023-05.io.nvda.nvme:0 --pf_id 0 --ctrl NVMeCtrl0 --suspended

# Create Namespace using the custom bdev (TestPT)
crictl exec $SNAP_ID snap_rpc.py nvme_namespace_create --nqn nqn.2023-05.io.nvda.nvme:0 --bdev_name TestPT --nsid 1

# Attach Namespace and Resume Controller
crictl exec $SNAP_ID snap_rpc.py nvme_controller_attach_ns --ctrl NVMeCtrl0 --nsid 1
crictl exec $SNAP_ID snap_rpc.py nvme_controller_resume --ctrl NVMeCtrl0

10.2. Manual Development Workflow

This approach sets up an interactive development container, ideal for workflows requiring frequent code changes, recompilation, and testing.

10.2.1. Environment Preparation

Prepare the working directory and clone sources (same as the Production method).

mkdir -p /opt/build
cd /opt/build
git clone https://github.com/Mellanox/spdk --branch v25.05.2.nvda

10.2.2. Create Development Dockerfile

Create a simplified Dockerfile that installs dependencies and provides a shell.

FROM nvcr.io/nvstaging/doca/doca_snap:4.9.0-6-doca3.2.0 as builder

RUN apt-get update && apt-get install -y \
    autoconf libtool python3-pyelftools libaio-dev \
    libncurses-dev libfuse3-dev patchelf libcmocka-dev make

ENTRYPOINT /bin/bash

10.2.3. Build and Run Interactive Container

1. Build the dev image:

   docker build -t doca_snap_custom_bdev_dev:latest -f Dockerfile .
   

2. Run interactively. Mount the necessary system volumes and your source code directory:

   docker run -ti --privileged --net=host \
       --volume /dev/hugepages:/dev/hugepages \
       --volume /dev/shm:/dev/shm \
       --volume /dev/infiniband:/dev/infiniband \
       --volume /etc/nvda_snap:/etc/nvda_snap \
       --volume ${PWD}/spdk/test/external_code:/external_code \
       doca_snap_custom_bdev_dev:latest
   

10.2.4. Build and Test Inside Container

Once inside the container shell, perform the following steps:

1. Compile the module:

   cd /external_code/
   export SPDK_HEADER_DIR=/opt/mellanox/spdk/include
   export SPDK_LIB_DIR=/opt/mellanox/spdk/lib
   export DPDK_LIB_DIR=/opt/mellanox/spdk/include
   
   make passthru_shared
   

2. Launch the service manually, preloading your compiled shared object:

   # Copy RPC definition
   cp /external_code/passthru/bdev_passthru.py /usr/bin/
   
   # Load environment variables
   source /opt/nvidia/nvda_snap/bin/set_environment_variables.sh
   
   # Start Service with LD_PRELOAD
   LD_PRELOAD=/external_code/passthru/libpassthru_external.so \
   /opt/nvidia/nvda_snap/bin/snap_service -m 0xff &
   

3. Configure and test:

   # Create base Malloc bdev
   spdk_rpc.py bdev_malloc_create 32 4096 -b Malloc0
   
   # Create custom Passthru bdev (explicitly specifying plugin)
   spdk_rpc.py --plugin bdev_passthru construct_ext_passthru_bdev -b Malloc0 -p TestPT
   
   # Setup NVMe Controller logic
   snap_rpc.py nvme_subsystem_create --nqn nqn.2023-05.io.nvda.nvme:0
   snap_rpc.py nvme_controller_create --nqn nqn.2023-05.io.nvda.nvme:0 --pf_id 0 --ctrl NVMeCtrl0 --suspended
   snap_rpc.py nvme_namespace_create --nqn nqn.2023-05.io.nvda.nvme:0 --bdev_name TestPT --nsid 1
   snap_rpc.py nvme_controller_attach_ns --ctrl NVMeCtrl0 --nsid 1
   snap_rpc.py nvme_controller_resume --ctrl NVMeCtrl0
   

Last updated: May 27, 2026