BlueField Scalable Functions

This page explains how to allocate resources for scalable functions (SFs), create and deploy an SF, use its functional network interface and representor, and remove the SF.

Overview

Scalable functions (SFs), also called subfunctions, are lightweight functions created from a parent PCI Express (PCIe) physical function (PF). Each SF has its own function capabilities, resources, and transmit and receive queues while sharing PCIe-level resources with its parent function and other SFs.

SFs can coexist with Single Root I/O Virtualization (SR-IOV) virtual functions (VFs), but SFs do not require PCIe SR-IOV to be enabled. SFs also support embedded switch (eSwitch) representors, allowing multiple services to use the networking device concurrently on the BlueField DPU.

The following diagram shows the relationship between SF management, the SF representor, and the functional SF network interface:

BlueField Arm system
+-- mlxdevm management interface
+-- SF representor --> eSwitch, OVS, or another forwarding configuration

Selected PCI controller
+-- Functional SF network interface --> workload or application

The SF representor is created on the BlueField Arm system. The selected controller determines where the functional SF network interface becomes available after the SF is activated and its driver is loaded.

Before You Begin

Requirements

Before configuring SFs:

• Install the required BlueField software. For installation instructions, refer to the DOCA Installation Guide for Linux.

• Use firmware version 20.30.1004 or later.

• Enable the following Linux kernel configuration options:

  • MLX5_ESWITCH

  • MLX5_SF

• Verify that mlxdevm is available at /opt/mellanox/iproute2/sbin/mlxdevm. The tool is supplied with the iproute2 package.

Choose the Target Controller

Unless otherwise specified, run the mlxdevm management commands on the BlueField Arm system. The optional controller parameter selects the PCI controller on which the functional SF device is instantiated.

In a standard single-host BlueField DPU system:

The SF representor is created on the BlueField Arm system in both cases. The netdev shown in mlxdevm port add output is the SF representor, not the functional SF network interface.

In a multi-host configuration, use mlxdevm port show to inspect controller assignments and consult the platform-specific topology documentation to identify the controller associated with the target host PF.

Understand the SF Identifiers

The procedure uses several identifiers. Record them as you create and deploy the SF.

Some mlxdevm commands accept either pci/<pci_address>/<sf_port_index> or the SF representor name. This page uses the PCI address and SF port index so that the identifier is explicit.

Configure SF Resources

Run the following command separately for each PF or embedded CPU physical function (ECPF) that will create SFs. The -d <pci_address> argument selects the function to configure. The same configuration parameter names apply to PFs and ECPFs.

mlxconfig -d 0000:03:00.0 set \
    PF_BAR2_ENABLE=0 \
    PER_PF_NUM_SF=1 \
    PF_TOTAL_SF=236 \
    PF_SF_BAR_SIZE=10

For example, PF_SF_BAR_SIZE=10 and PF_TOTAL_SF=14 allocate 14 MB of BAR2 in total.

Create and Deploy an SF

The SF lifecycle consists of creating the SF port, configuring and activating the SF, and deploying the functional driver. The following procedure separates those lifecycle stages into discrete tasks so that each identifier and execution environment is clear.

Create SF port
    --> Configure and activate SF
        --> Identify auxiliary device
            --> Bind functional driver
                --> Verify and use SF

Unless otherwise specified, run the commands in this section on the BlueField Arm system.

Create the SF Port

Create an SF by adding a port with the pcisf flavor:

/opt/mellanox/iproute2/sbin/mlxdevm port add pci/<pci_address> flavour pcisf pfnum <pf_number> sfnum <sf_number>

Each SF must have a unique SF number. SF numbers 1000 and later are reserved for the virtio-net controller.

The following example creates SF 4 on PF 0 of the local BlueField Arm-side controller. Because the controller argument is omitted, controller 0 is used:

/opt/mellanox/iproute2/sbin/mlxdevm port add pci/0000:03:00.0 flavour pcisf pfnum 0 sfnum 4

Illustrative output:

pci/0000:03:00.0/229409: type eth netdev en3f0pf0sf4 flavour pcisf controller 0 pfnum 0 sfnum 4
    function:
      hw_addr 00:00:00:00:00:00 state inactive opstate detached roce true max_uc_macs 128 trust off

Record the following values from the output:

• SF number: 4

• SF port index: 229409

• SF representor: en3f0pf0sf4

• Controller: 0

To inspect all created SF ports and their current attributes, run:

/opt/mellanox/iproute2/sbin/mlxdevm port show

To create an SF on PF 0 of the external host controller, specify controller 1:

/opt/mellanox/iproute2/sbin/mlxdevm port add pci/0000:03:00.0 flavour pcisf pfnum 0 sfnum 88 controller 1

Example output:

pci/0000:03:00.0/32768: type eth netdev eth6 flavour pcisf controller 1 pfnum 0 sfnum 88 splittable false
    function:
      hw_addr 00:00:00:00:00:00 state inactive opstate detached

In this output, eth6 is the SF representor on the BlueField Arm system. It is not the functional SF network interface on the external host.

Configure and Activate the SF

Configure the SF through its management interface before binding the functional driver. The following command sets the MAC address, enables trust mode, and activates the SF:

/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/<pci_address>/<sf_port_index> \
    hw_addr <mac_address> \
    trust on \
    state active

For example:

/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/0000:03:00.0/229409 \
    hw_addr 02:25:f2:8d:a2:4c \
    trust on \
    state active

A trusted function has additional privileges, such as the ability to update the steering database.

Set all required function attributes before binding the functional SF driver.

Identify the Auxiliary Device

After activating an SF created on controller 0, identify the auxiliary device that corresponds to the SF before binding its functional driver.

List the available devices:

devlink dev show

Example output after creating and activating the SF:

pci/0000:03:00.0
pci/0000:03:00.1
auxiliary/mlx5_core.sf.2
auxiliary/mlx5_core.sf.3
auxiliary/mlx5_core.sf.4

The numeric suffix in auxiliary/mlx5_core.sf.<serial> is the auxiliary-device serial. Do not assume that this serial is the same as the SF number. Match the auxiliary device to the SF by reading its sfnum file:

cat /sys/bus/auxiliary/devices/mlx5_core.sf.<serial>/sfnum

For example:

cat /sys/bus/auxiliary/devices/mlx5_core.sf.4/sfnum

Example output:

4

This output confirms that mlx5_core.sf.4 corresponds to SF number 4.

Bind the SF Driver

Unbind the SF from the default configuration driver and bind it to the functional SF driver:

echo mlx5_core.sf.<serial> > /sys/bus/auxiliary/drivers/mlx5_core.sf_cfg/unbind
echo mlx5_core.sf.<serial> > /sys/bus/auxiliary/drivers/mlx5_core.sf/bind

For example:

echo mlx5_core.sf.4 > /sys/bus/auxiliary/drivers/mlx5_core.sf_cfg/unbind
echo mlx5_core.sf.4 > /sys/bus/auxiliary/drivers/mlx5_core.sf/bind

Verify the SF

Run the following commands on the BlueField Arm system:

/opt/mellanox/iproute2/sbin/mlxdevm port show
devlink dev show
ip link show

Verify the following:

• mlxdevm port show lists the expected SF number, controller, representor, MAC address, trust mode, and state.

• devlink dev show lists the auxiliary device matched to the SF.

• The SF representor exists on the BlueField Arm system.

• For an SF created on controller 0, the functional SF network interface exists on the BlueField Arm system after driver binding.

• For an SF created on controller 1, verify the functional network interface on the external host after completing the host-side deployment procedure documented for the target software release.

Use an SF

The functional SF network interface and the SF representor serve different purposes.

Do not configure the workload IP address on the SF representor.

Example: Run an Application Using Two SFs and OVS

This example creates two SFs on controller 0, connects their representors to two OVS bridges, and initializes their auxiliary devices in a DOCA application. Run all commands in this example on the BlueField Arm system.

The example creates the following forwarding paths:

p0      <--> sf_bridge1 <--> en3f0pf0sf4
pf0hpf  <--> sf_bridge2 <--> en3f0pf0sf5

Two SFs associated with the same PCIe function are required for this example.

Prepare SFs 4 and 5

Create the SF ports:

/opt/mellanox/iproute2/sbin/mlxdevm port add pci/0000:03:00.0 flavour pcisf pfnum 0 sfnum 4
/opt/mellanox/iproute2/sbin/mlxdevm port add pci/0000:03:00.0 flavour pcisf pfnum 0 sfnum 5

Run the following command and record the SF port index and representor name returned for each SF:

/opt/mellanox/iproute2/sbin/mlxdevm port show

Configure and activate the two SFs by using the actual SF port indices returned by the command:

/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/0000:03:00.0/<sf4_port_index> \
    hw_addr 02:25:f2:8d:a2:4c trust on state active

/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/0000:03:00.0/<sf5_port_index> \
    hw_addr 02:25:f2:8d:a2:5c trust on state active

For each SF, complete the steps in Identify the Auxiliary Device and Bind the SF Driver. Record the auxiliary-device serial assigned to each SF.

Remove an Existing Bridge, if Necessary

This optional example removes an existing bridge named ovsbr1:

ovs-vsctl del-br ovsbr1

Create the OVS Bridges

Create sf_bridge1 and sf_bridge2, and add the physical-port and host-PF representors:

ovs-vsctl add-br sf_bridge1
ovs-vsctl add-br sf_bridge2
ovs-vsctl add-port sf_bridge1 p0
ovs-vsctl add-port sf_bridge2 pf0hpf

Add the SF Representors

Add each SF representor to its bridge:

ovs-vsctl add-port sf_bridge1 en3f0pf0sf4
ovs-vsctl add-port sf_bridge2 en3f0pf0sf5

If either representor is down, bring it up:

ip link set en3f0pf0sf4 up
ip link set en3f0pf0sf5 up

Verify the OVS Configuration

Run:

ovs-vsctl show

Example output:

Bridge sf_bridge1
    Port p0
        Interface p0
    Port sf_bridge1
        Interface sf_bridge1
            type: internal
    Port en3f0pf0sf4
        Interface en3f0pf0sf4
Bridge sf_bridge2
    Port sf_bridge2
        Interface sf_bridge2
            type: internal
    Port en3f0pf0sf5
        Interface en3f0pf0sf5
    Port pf0hpf
        Interface pf0hpf
ovs_version: "2.14.1"

Run the Application

Initialize the two SF auxiliary devices when starting the application:

<executable_binary> \
    -a auxiliary:mlx5_core.sf.<sf4_serial> \
    -a auxiliary:mlx5_core.sf.<sf5_serial>

For example, if the assigned auxiliary-device serials are 4 and 5:

doca_<app_name> \
    -a auxiliary:mlx5_core.sf.4 \
    -a auxiliary:mlx5_core.sf.5 \
    -- [application_flags]

Manage SFs

Unless otherwise specified, run the commands in this section on the BlueField Arm system.

List and Inspect SFs

To list SF ports and function attributes, run:

/opt/mellanox/iproute2/sbin/mlxdevm port show

To list PCI and SF auxiliary devices, run:

devlink dev show

To identify which SF number corresponds to an auxiliary device, run:

cat /sys/bus/auxiliary/devices/mlx5_core.sf.<serial>/sfnum

Remove an SF

If the SF representor is connected to an OVS bridge, remove it from the bridge first:

ovs-vsctl del-port <bridge_name> <sf_representor>

Set the SF state to inactive:

/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/<pci_address>/<sf_port_index> state inactive

Delete the SF port:

/opt/mellanox/iproute2/sbin/mlxdevm port del pci/<pci_address>/<sf_port_index>

For example:

ovs-vsctl del-port sf_bridge1 en3f0pf0sf4
/opt/mellanox/iproute2/sbin/mlxdevm port function set pci/0000:03:00.0/229409 state inactive
/opt/mellanox/iproute2/sbin/mlxdevm port del pci/0000:03:00.0/229409

Use devlink on an Upstream Kernel

For an upstream-based kernel on which mlxdevm is unavailable, follow the Upstream Guide on Scalable Functions to perform the equivalent operations with devlink.

Related Documentation

• DOCA Installation Guide for Linux

• NVIDIA BlueField Reset and Reboot Procedures

• BlueField Virtio-net Controller

• Upstream Guide on Scalable Functions

Last updated: August 03, 2026