The tool can support new devices only once the tool is upgraded to its latest version.
mlxfwstress enables/disables various firmware stress flows. It can work in multiple modes:
-
Enable/disable a specific set of stress types
-
Clear all stress types
-
Random mode:
-
Single mode - choose one stress type in each iteration and enable/disable it
-
Wild-mode- choose multiple stress types in each iteration and enable/disable them
-
Each time a stress type is chosen in a random iteration, the opposite operation is done on it (e.g., if a stress type is turned on, in the next iteration it will be turned off and vice versa).
-
Toggle mode:
-
Turns on and off the list of stress types alternating. Can be used with iterations.
To disable a stressor while in toggling mode, first you must disable the mlxfwstress tool, and only after that disable the stressor.
-
-
Clear semaphore:
mlxfwstress Synopsis
# mlxfwstress [-d|--dev <DeviceName>] [-h|--help] [-v|--version] [-o|--operation <Operation>] [--rand-mode <Random mode>] [-t|--stress-type <Stress type>] [--iterations <Iterations>] [--stress-delay <Stress delay>] [--max-rand-on <Max rand on>] [--hang-type <Hang type>] [--seed <seed>] [--toggle-time <x,y>]
where:
|
-d|--dev <DeviceName> |
Perform operation for a specified device |
|
-h|--help |
Show this message and exit |
|
-v|--version |
Show the executable version and exit |
|
-o|--operation <Operation> |
Choose operation: on, off, clear_all, random query, clear_semaphore |
|
--rand-mode <Random mode> |
Choose a random mode: single, wild |
|
-t|--stress-type <Stress type> |
Specify a list of stress types separated by comma. (See Stress Types.) |
|
--iterations <Iterations> |
Specify the number of iterations. |
|
--stress-delay <Stress delay> |
Specify the stress delay in seconds (can be float).
Recommended values: 0-1 |
|
--max-rand-on <Max rand on> |
Specify the maximal time a stress is allowed to be on in random mode in seconds. Recommended values (0,1] Default is 1 |
|
--hang-type <Hang type> |
Specify a list of hang types separated by comma. (See Hang Types.) |
|
--seed <seed> |
Specify the seed for the random. |
|
--toggle-time <x,y> |
Toggle time after off, both in seconds (can be float). If y is not supplied the tool will use equal values for x and y |
Turning On Stress Types
To turn on a specific stress type:
mlxfwstress -d /dev/mst/mt4129_pciconf0 -o on -t INVALIDATE_INTERNAL_CACHE_RX_1
-------------------------------------------------
Operation: [ON]
-------------------------------------------------
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED
To turn on a set of stress types:
mlxfwstress -d /dev/mst/mt4129_pciconf0 -o on -t INVALIDATE_INTERNAL_CACHE_RX_1,INVALIDATE_INTERNAL_CACHE_RX_2
-------------------------------------------------
Operation: [ON]
-------------------------------------------------
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -PASSED
To turn on all the available stress types:
mlxfwstress -d mt4119_pciconf0 -t ALL -o on
Random seed: [1587969653]
-------------------------------------------------
Operation: [ON]
-------------------------------------------------
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_CQ -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_GENERAL_1 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_QP -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_3 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_4 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_RX_5 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX1 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX2 -PASSED
Turning ON stress type: INVALIDATE_INTERNAL_CACHE_SX3 -PASSED
Turning Off Stress Types
To turn off a specific stress type:
mlxfwstress -d mt4129_pciconf0 -o off -t INVALIDATE_INTERNAL_CACHE_RX_1
-------------------------------------------------
Operation: [OFF]
-------------------------------------------------
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED
To turn off a set of stress types:
mlxfwstress -d mt4129_pciconf0 -o off -t INVALIDATE_INTERNAL_CACHE_RX_1,INVALIDATE_INTERNAL_CACHE_RX_2
-------------------------------------------------
Operation: [OFF]
-------------------------------------------------
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_1-PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_2-PASSED
Querying the Stress Types
To query the state of all stress types:
mlxfwstress -d mt4117_pciconf0 -o query -t ALL
-------------------------------------------------
Operation: [QUERY]
-------------------------------------------------
Querying stress type: INVALIDATE_INTERNAL_CACHE_CQ -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_GENERAL_1 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_QP -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_3 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_4 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_RX_5 -NOT SUPPORTED
Querying stress type: INVALIDATE_INTERNAL_CACHE_SX1 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_SX2 -ENABLED
Querying stress type: INVALIDATE_INTERNAL_CACHE_SX3 -ENABLED
Hang Types
ConnectX-4/ConnectX-4 Lx/ConnectX-5/ConnectX-6/ConnectX-6Lx/ConnectX-7/ConnectX-8 Adapter Cards Hang Types
The following are the hang types available for ConnectX-4/ConnectX-4 Lx/ConnectX-5/ConnectX6/ConnectX-6lx/ConnectX-7/ConnectX-8 adapter cards:
|
Category |
Stress Type |
Description |
Notes |
|---|---|---|---|
|
Hang FW/HW |
FFSER |
Initialize FaultInjector object |
|
|
STOP_RX_PER_PRIO1 |
|
This type requires the following extra flags:
|
mlxfwstress -d mt4115_pciconf0 -o on --hang-type STOP_RX_PER_PRIO --extra %STOP_RX_PER_PRIO[0x00100FF]
Random seed: [1588056318]
-------------------------------------------------
Operation: [ON]
-------------------------------------------------
Turning ON hang type: STOP_RX_PER_PRIO -PASSED
To turn this Hang Type, the command must be executed in the following format:
Example:
mlxfwstress -d mt4115_pciconf0 -o on --hang-type STOP_RX_PER_PRIO --extra % STOP_RX_PER_PRIO [0x000100FF]
output:
Random seed: [1573642282]
-------------------------------------------------
Operation: [ON]
-------------------------------------------------
Turning ON hang type: STOP_RX_PER_PRIO-PASSED
Querying the Hang Types
To query the state of all hang types:
mlxfwstress -d mt4103_pciconf0 -o query --hang-type ALL
-------------------------------------------------
Operation: [QUERY]
-------------------------------------------------
Querying hang type: Sx1 -ENABLED
Querying hang type: Rx1 -ENABLED
Querying hang type: Tx -ENABLED
Querying hang type: Rx -ENABLED
Clearing all Stress/Hang Types
To clear all stress/hang types:
mlxfwstress -d mt4129_pciconf0 -o clear_all
-------------------------------------------------
Operation: [CLEAR_ALL]
-------------------------------------------------
Turning OFF hang type: FFSER -NOT SUPPORTED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_CQ -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_GENERAL_1 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_QP -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_1 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_2 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_3 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_4 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_RX_5 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_SX1 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_SX2 -PASSED
Turning OFF stress type: INVALIDATE_INTERNAL_CACHE_SX3 -PASSED
Random Operation
There are two random modes you can choose from:
-
Single - gives a set of stress types, in each iteration one stress type is chosen an toggled ON/OFF according to his current state
-
Wild - gives a set of stress types, in each iteration a random subset of stress types is chosen and toggled ON/OFF according to their current state
Setting the Random Mode for the Stress Types
To set the Single Mode:
mlxfwstress -d mt4103_pciconf0 -o random --rand-mode single -t STOP_CE_INSTAGE_EQE --stress-delay 0.2 --iterations 10
-------------------------------------------------
Operation: [RANDOM]
-------------------------------------------------
#############################################
Random:
Iterations delay: 0.2 [sec]
Iterations number: 10
Max on time: 1 [sec]
#############################################
RANDOM ITERATION: [1]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 0 [ms]
RANDOM ITERATION: [2]
[stop_ce_instage_eqe]: [OFF], duration since last operation: 200 [ms]
RANDOM ITERATION: [3]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 201 [ms]
RANDOM ITERATION: [4]
[stop_ce_instage_eqe]: [OFF], duration since last operation: 200 [ms]
RANDOM ITERATION: [5]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms]
RANDOM ITERATION: [6]
[stop_ce_instage_eqe]: [OFF], duration since last operation: 201 [ms]
RANDOM ITERATION: [7]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms]
RANDOM ITERATION: [8]
[stop_ce_instage_eqe]: [OFF], duration since last operation: 201 [ms]
RANDOM ITERATION: [9]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms]
Turning OFF stress type: stop_ce_instage_eqe
RANDOM ITERATION: [10]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 200 [ms]
=======================================================
Turning off all stress types after random:
Turning OFF stress type: stop_ce_instage_eqe
-
As seen in the example above, after the specified number of iterations, the tool turns off all the stress types.
-
The default value for stress-delay is 1 second.
-
If no number of iterations was supplied then the user is expected to stop the tool with ctrl+c. The tool turns off all the stress types.
To set the Wild Mode:
mlxfwstress -d mt4103_pciconf0 -o random --rand-mode wild -t ALL --stress-delay 0.2 --max-rand-on 1 --iterations 5
-------------------------------------------------
Operation: [RANDOM]
-------------------------------------------------
#############################################
Random:
Iterations delay: 0.2 [sec]
Iterations number: 5
Max on time: 1 [sec]
#############################################
RANDOM ITERATION: [1]
[stop_ce_instage_eqe]: [ON] , duration since last operation: 0 [ms]
[stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 0 [ms]
[stop_edbh]: [ON] , duration since last operation: 0 [ms]
[stop_idbh]: [ON] , duration since last operation: 0 [ms]
[stop_qpc_miss_machine_0]: [ON] , duration since last operation: 0 [ms]
[stop_qpc_miss_machine_3]: [ON] , duration since last operation: 0 [ms]
[lock_cegw]: [ON] , duration since last operation: 0 [ms]
[lock_obgw_tcu]: [ON] , duration since last operation: 0 [ms]
[lock_qpcgw_rx]: [ON] , duration since last operation: 0 [ms]
[lock_semaphore_ipc_sx1]: [ON] , duration since last operation: 0 [ms]
RANDOM ITERATION: [2]
[stop_sxp_vl_arb_port1]: [ON] , duration since last operation: 0 [ms]
[stop_edbh]: [OFF], duration since last operation: 203 [ms]
[stop_idbh]: [OFF], duration since last operation: 203 [ms]
[stop_qpc_miss_machine_3]: [OFF], duration since last operation: 202 [ms]
[lock_cegw]: [OFF], duration since last operation: 202 [ms]
[lock_obgw_tpt]: [ON] , duration since last operation: 0 [ms]
[lock_obgw_tcu]: [OFF], duration since last operation: 203 [ms]
[lock_semaphore_ipc_rx0]: [ON] , duration since last operation: 0 [ms]
[lock_semaphore_ipc_rx1]: [ON] , duration since last operation: 0 [ms]
[lock_semaphore_ipc_ldb]: [ON] , duration since last operation: 0 [ms]
RANDOM ITERATION: [3]
[stop_ce_instage_eqe]: [OFF], duration since last operation: 406 [ms]
[stop_sxp_vl_arb_port2]: [OFF], duration since last operation: 406 [ms]
[stop_edbh]: [ON] , duration since last operation: 203 [ms]
[stop_idbh]: [ON] , duration since last operation: 203 [ms]
[stop_qpc_miss_machine_0]: [OFF], duration since last operation: 406 [ms]
[stop_qpc_miss_machine_2]: [ON] , duration since last operation: 0 [ms]
[lock_obgw_tpt]: [OFF], duration since last operation: 203 [ms]
[lock_obgw_sxd]: [ON] , duration since last operation: 0 [ms]
[lock_semaphore_ipc_sx1]: [OFF], duration since last operation: 405 [ms]
[lock_semaphore_ipc_ldb]: [OFF], duration since last operation: 203 [ms]
RANDOM ITERATION: [4]
[stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 203 [ms]
[stop_edbh]: [OFF], duration since last operation: 202 [ms]
[stop_idbh]: [OFF], duration since last operation: 202 [ms]
[stop_qpc_miss_machine_1]: [ON] , duration since last operation: 0 [ms]
[stop_qpc_miss_machine_3]: [ON] , duration since last operation: 406 [ms]
[lock_obgw_tpt]: [ON] , duration since last operation: 202 [ms]
[lock_obgw_tcu]: [ON] , duration since last operation: 406 [ms]
[lock_obgw_sxd]: [OFF], duration since last operation: 203 [ms]
[lock_semaphore_ipc_sx1]: [ON] , duration since last operation: 203 [ms]
[lock_semaphore_ipc_rx1]: [OFF], duration since last operation: 406 [ms]
[invalidate_caches]: [ON] , duration since last operation: 0 [ms]
Turning OFF stress type: stop_sxp_vl_arb_port1
Turning OFF stress type: stop_sxp_vl_arb_port2
Turning OFF stress type: stop_qpc_miss_machine_1
Turning OFF stress type: stop_qpc_miss_machine_2
Turning OFF stress type: stop_qpc_miss_machine_3
Turning OFF stress type: lock_obgw_tpt
Turning OFF stress type: lock_obgw_tcu
Turning OFF stress type: lock_qpcgw_rx
Turning OFF stress type: lock_semaphore_ipc_sx1
Turning OFF stress type: lock_semaphore_ipc_rx0
Turning OFF stress type: invalidate_caches
RANDOM ITERATION: [5]
[stop_sxp_vl_arb_port2]: [ON] , duration since last operation: 202 [ms]
[stop_idbh]: [ON] , duration since last operation: 322 [ms]
[lock_obgw_tpt]: [ON] , duration since last operation: 202 [ms]
[lock_obgw_tcu]: [ON] , duration since last operation: 202 [ms]
[lock_qpcgw_rx]: [ON] , duration since last operation: 202 [ms]
[invalidate_caches]: [ON] , duration since last operation: 202 [ms]
=======================================================
Turning off all stress types after random:
Turning OFF stress type: stop_sxp_vl_arb_port2
Turning OFF stress type: stop_idbh
Turning OFF stress type: lock_obgw_tpt
Turning OFF stress type: lock_obgw_tcu
Turning OFF stress type: lock_qpcgw_rx
Turning OFF stress type: invalidate_caches
Last updated: