Networking NVIDIA BlueField BSP
Networking NVIDIA BlueField BSP
Breadcrumbs

QoS Configuration

To learn more about port QoS configuration, refer to this community post.

When working in Embedded Host mode, using mlnx_qos on both the host and Arm will result with undefined behavior. Users must only use mlnx_qos from the Arm. After changing the QoS settings from Arm, users must restart the mlx5 driver on host.

Rate Limiting Host PF and VF

From the Arm, users may limit the transmit rate of the PF in the host. For example, these commands limit the transmit rate of the PF of ECPF's p0 to 1000 Mbps and the virtual function #0 to 500 Mbps:

echo 1000 > /sys/class/net/p0/smart_nic/pf/max_tx_rate
echo 500 > /sys/class/net/p0/smart_nic/vf0/max_tx_rate

To set the minimum transmit rate of the PF of ECPF's p0 to 100 Mbps and of VF0 to 50 Mbps, run the following commands:

echo 100 > /sys/class/net/p0/smart_nic/pf/min_tx_rate"
echo 50 > /sys/class/net/p0/smart_nic/vf0/min_tx_rate"

The minimum transmit rate is implemented according to weight setting and round robin arbitration. Thus, this configuration only ensures that the minimum transmit rate of the PF is double that of VF0.

Rate Limiting SF

This section explains how to configure QoS group and SF QoS settings using mlxdevm, which located under directory /opt/mellanox/iproute2/sbin/.

The settings of a QoS group include creating/deleting a QoS group and modifying its tx_max and tx_share values. The settings of SF QoS include modifying its tx_max and tx_share values, assigning an SF to a QoS group, and unassigning an SF from a QoS group. This section focuses on the configuration syntax.

Please refer to section "Limit and Bandwidth Share Per VF" in the MLNX_OFED User Manual for detailed explanation on vPort QoS behaviors.

mlxdevm Commands

mlxdevm port function rate add


mlxdevm port function rate add <DEV>/<GROUP_NAME>

Adds a QoS group.

Syntax Description

DEV/GROUP_NAME

Specifies group name in string format

Example

This command adds a new QoS group named "12_group" under device "pci/0000:03:00.0":


mlxdevm port function rate add pci/0000:03:00.0/12_group

Notes


mlxdevm port function rate del


mlxdevm port function rate del <DEV>/<GROUP_NAME>

Deletes a QoS group.

Syntax Description

DEV/GROUP_NAME

Specifies group name in string format

Example

This command deletes QoS group "12_group" from device "pci/0000:03:00.0":


mlxdevm port function rate del pci/0000:03:00.0/12_group

Notes


mlxdevm port function rate set tx_max tx_share


mlxdevm port function rate set {<DEV>/<GROUP_NAME> | <DEV>/<PORT_INDEX>} tx_max <TX_MAX> [tx_share <TX_SHARE>]

Sets tx_max and tx_share for QoS group or mlxdevm port.

Syntax Description

DEV/GROUP_NAME

Specifies the group name to operate on

DEV/PORT_INDEX

Specifies the mlxdevm port to operate on

TX_MAX

tx_max bandwidth in Mb/s

TX_SHARE

tx_share bandwidth in Mb/s

Example

This command sets tx_max to 900Mb/s for the "12_group" QoS group: 

mlxdevm port function rate set pci/0000:03:00.0/12_group tx_max 900

This command sets tx_max to 2000Mb/s and tx_share to 500Mb/s for this function:

mlxdevm port function rate set pci/0000:03:00.0/229376 tx_max 2000 tx_share 500

Notes


mlxdevm port function rate set parent


mlxdevm port function rate set <DEV>/<PORT_INDEX> {parent <PARENT_GROUP_NAME>}

Assigns mlxdevm port to a QoS group.

Syntax Description

DEV/PORT_INDEX

Specifies the mlxdevm port to operate on

PARENT_GROUP_NAME

parent group name in string format

Example

This command assigns this function to the QoS group "12_group":

mlxdevm port function rate set pci/0000:03:00.0/229376 parent 12_group

Notes


mlxdevm port function rate set noparent


mlxdevm port function rate set <DEV>/<PORT_INDEX> noparent

Ungroups a mlxdevm port.

Syntax Description

DEV/PORT_INDEX

Specifies the mlxdevm port to operate on

Example

This command ungroups this function:


mlxdevm port function rate set pci/0000:03:00.0/229376 noparent

Notes


mlxdevm port function rate show


mlxdevm port function rate show [<DEV>/<GROUP_NAME> | <DEV>/<PORT_INDEX>]

Displays QoS information QoS group or mlxdevm port.

Syntax Description

DEV/GROUP_NAME

Specifies the group name to display

DEV/PORT_INDEX

Specifies the mlxdevm port to display

Example

This command displays the QoS info of all QoS groups and mlxdevm ports on the system: 

mlxdevm port function rate show
pci/0000:03:00.0/12_group type node tx_max 1000 tx_share 200
pci/0000:03:00.0/229376 type leaf tx_max 2000 tx_share 500 parent 12_group

This command displays QoS info of 12_group: 

mlxdevm port function rate show pci/0000:03:00.0/12_group
pci/0000:03:00.0/12_group type node tx_max 1000 tx_share 200

Notes

If a QoS group name or mlxdevm port are not specified, all QoS groups and mlxdevm ports are displayed.

Rate Limiting VF Group

This section explains how to configure QoS group and VF QoS settings using devlink located under /opt/mellanox/iproute2/sbin/.

The settings of a QoS group include creating/deleting a QoS group and modifying its tx_max and tx_share values. The settings of VF QoS include modifying its tx_max and tx_share values, assigning a VF to a QoS group, and unassigning a VF from a QoS group. This section focuses on the configuration syntax.

Please refer to section "Limit and Bandwidth Share Per VF" in the MLNX_OFED User Manual for detailed explanation on vPort QoS behaviors.

devlink port function rate add <DEV>/<GROUP_NAME>

Adds a QoS group.

Syntax Description

DEV/GROUP_NAME

Specifies group name in string format

Example

This command adds a new QoS group named "12_group" under device "pci/0000:03:00.0":


devlink port function rate add pci/0000:03:00.0/12_group

Notes



devlink port function rate del <DEV>/<GROUP_NAME>

Deletes a QoS group.

Syntax Description

DEV/GROUP_NAME

Specifies group name in string format

Example

This command deletes QoS group "12_group" from device "pci/0000:03:00.0":


devlink port function rate del pci/0000:03:00.0/12_group

Notes



devlink port function rate set {<DEV>/<GROUP_NAME> | <DEV>/<PORT_INDEX>} tx_max <TX_MAX> [tx_share <TX_SHARE>]

Sets tx_max and tx_share for QoS group or devlink port.

Syntax Description

DEV/GROUP_NAME

Specifies the group name to operate on

DEV/PORT_INDEX

Specifies the devlink port to operate on

TX_MAX

tx_max bandwidth in Mb/s

TX_SHARE

tx_share bandwidth in Mb/s

Example

This command sets tx_max to 2000Mb/s and tx_share to 500Mb/s for the "12_group" QoS group: 

devlink port function rate set pci/0000:03:00.0/12_group tx_max 2000Mbps tx_share 500Mbps

This command sets tx_max to 2000Mb/s and tx_share to 500Mb/s for the VF represented by port index 196609:

devlink port function rate set pci/0000:03:00.0/196609 tx_max 200Mbps tx_share 50Mbps

This command displays a mapping between VF devlink ports and netdev names:


$ devlink port

In the output of this command, VFs are indicated by flavour pcivf.

Notes



devlink port function rate set <DEV>/<PORT_INDEX> {parent <PARENT_GROUP_NAME>}

Assigns devlink port to a QoS group.

Syntax Description

DEV/PORT_INDEX

Specifies the devlink port to operate on

PARENT_GROUP_NAME

parent group name in string format

Example

This command assigns this function to the QoS group "12_group":

devlink port function rate set pci/0000:03:00.0/196609 parent 12_group

Notes



devlink port function rate set <DEV>/<PORT_INDEX> noparent

Ungroups a devlink port.

Syntax Description

DEV/PORT_INDEX

Specifies the devlink port to operate on

Example

This command ungroups this function:


devlink port function rate set pci/0000:03:00.0/196609 noparent

Notes



devlink port function rate show [<DEV>/<GROUP_NAME> | <DEV>/<PORT_INDEX>]

Displays QoS information QoS group or devlink port.

Syntax Description

DEV/GROUP_NAME

Specifies the group name to display

DEV/PORT_INDEX

Specifies the devlink port to display

Example

This command displays the QoS info of all QoS groups and devlink ports on the system: 

devlink port function rate show
pci/0000:03:00.0/12_group type node tx_max 2000Mbps tx_share 500Mbps
pci/0000:03:00.0/196609 type leaf tx_max 200Mbps tx_share 50Mbps parent 12_group

This command displays QoS info of 12_group: 

devlink port function rate show pci/0000:03:00.0/12_group
pci/0000:03:00.0/12_group type node tx_max 2000Mbps tx_share 500Mbps

Notes

If a QoS group name or devlink port are not specified, all QoS groups and devlink ports are displayed.

Last updated: