The mlxlink tool is used to check and debug link status and related issues. The tool can be used on different links and cables (passive, active, transceiver and backplane).
-
In order for mlxlink to function properly, make sure to update the firmware version to the latest version
-
mlxlink is intended for advanced users with appropriate technical background
-
When using mlxlink to disable the port state ("--port_state dn" flag) on a NIC connected through a Socket-Direct connection, the port must be disabled on both mst devices representing the physical port
-
Do not use mlxlink to disable the port connecting between the host and the unmanaged switch using (“--port_state dn”) flag
-
When running mlxlink to show SLTP for 16nm technology, the “--advanced” flag should be added to the run command
-
mlxlink errors, warnings and notes are printed on stderr console
-
Setting the speeds (50GbE, 100GbE and 200GbE) for the new devices (NVIDIA ConnectX-6 and above, NVIDIA Quantum switches and above) requires specifying the number of lanes for the speed: mlxlink -d <dev> --speeds [50G_2X | 50G_1X | 100G_2X | 100G_4X | 200G_4X]
-
In ConnectX-7 and later cards, configuring the loopback can be applied when the link is fully down (not in polling state)
-
mlxlink is a diagnostic tool and is not intended for continuous polling or background monitoring
mlxlink Usage
To run mlxlink:
mlxlink [OPTIONS]
where:
Options:
|
-h |--help |
Display help message. |
|
-v |--version |
Display version info. |
|
-d |--device <device> |
Perform operation for a specified mst device |
|
-p |--port <port_number> |
Port Number |
|
--port_type <port_type> |
Port Type [NETWORK(Default)/PCIE/OOB] |
|
--depth <depth> |
Depth level of the DUT of some hierarchy (valid for PCIe port type only) |
|
--pcie_index <pcie_index> |
PCIe index number (Internal domain index) (valid for PCIe port type only) |
|
--node <node> |
The node within each depth (valid for PCIe port type only) |
|
--json |
Print the output in JSON format |
|
--linkx_els |
Show ELS module info on a CPO system. Must be used with -m / --show_module. Mutually exclusive with --linkx_oe. Not valid on a non-CPO device. On an Ethernet virtual module the flag is ignored. |
|
--linkx_oe |
Show OE module info on a CPO system. Must be used with -m / --show_module. Mutually exclusive with --linkx_els. Not valid on a non-CPO device, and not supported on an Ethernet virtual module. |
|
--plane <plane> |
Plane to access on an HCA that supports planarization. Index starts at 1. Not valid with --show_eye, --show_serdes_tx, --show_prr, --test_mode, or --serdes_tx. |
Queries:
|
--show_links |
Show valid PCIe links (valid for PCIe port type only) and the link status |
|
-m |--show_module |
Show Module Info |
|
-c |--show_counters |
Show Physical Counters and BER Info |
|
-e |--show_eye |
Show Eye Opening Info |
|
--eye_lanes <lanes> |
Show eye info for a subset of lanes, comma-separated, for example 0,2,3. Optional. Used with --show_eye. This is not --lane. |
|
--show_fec |
Show FEC Capabilities |
|
--show_serdes_tx |
Show Transmitter Info |
|
--show_tx_group_map <group_num> |
Display all label ports mapped to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices). |
|
--show_device |
General Device Info |
|
--show_ber_monitor |
Show BER Monitor Info. Note: The flag is not supported in HCAs. |
|
--show_external_phy |
Show External PHY Info Note: The flag is supported in NVIDIA Spectrum switch systems only. |
|
--show_plr |
Show Physical Layer Retransmission (PLR) Info Note: The flag is supported in NVL5 (and above) switch systems only. |
|
--show_apsu |
Show APSU (ILT/RTS) Info |
|
--show_kr |
Show Link Training Optimization (KR) Info
|
|
--show_rx_recovery_counters |
Show Rx Recovery Counters Note: The flag is supported in NVL5 (and above) switch systems only. |
|
--show_peq |
Show Link Periodic Equalization (PEQ) Info Note: The flag is supported in NVL5 (and above) switch systems only. |
|
--show_multi_port_info |--smpi |
Show Multi Port Info Table |
|
--show_multi_port_module_info |--smpmi |
Show Multi Port Module Info Table |
|
--show_multi_port_cpo_info | --smpci |
Show Multi Port Module CPO Info Table |
|
--show_bkv_groups |
Show BKV Groups Info (requires --lane parameter) |
|
--show_bkv_group <group_id> |
Show BKV Group Info (requires --lane parameter) |
|
--show_host_class |
Show host class information |
|
--show_prr |
Run PRR measurement on the given lane and dump meas_data (destructive: a link toggle is required afterwards; requires --lane) |
|
--show_phy_info |
Show PHY Info: per-lane RX signal detect and RX alignment-marker (AM) lock, live and as latched at the last link-down. |
|
--show_ltx |
Show LTX status and logger history Note: The flag is supported on NVLink ports only (GPU and NVSwitch). |
Commands:
|
-a |--port_state <port_state> |
Configure Port State [UP(up)/DN(down)/TG(toggle)] |
|
|
|
|
|
--all |
Configure Port State for all the links at once |
|
|
|
--module_state <module_state> |
Configure Module State [UP(up)/DN(down)/TG(toggle)] for the port's module. With --els_module instead of -p, it applies to that ELS module directly. |
|
|
|
|
-s |--speeds <speeds> |
Configure Speeds [XDR, NDR, HDR, EDR, FDR10, FDR, QDR, DDR, SDR, 1600G_8X, 800G_4X, 400G_2X, 200G_1X, 800G_8X, 400G_4X, 200G_2X, 100G_1X, 400G_8X, 200G_4X, 100G_2X, 50G_1X, 100G, 100G_4X, 50G, 50G_2X, 25G, 40G, 10G, 2.5G, 1G, NVLink-XDR_1X, 400G_2X_MODE_B, 360G_2X_MODE_B, 328G_2X_MODE_B, 378G_2X_MODE_B, 345G_2X_MODE_B, 187G_1X_MODE_A, 176G_1X_MODE_A, 169G_1X_MODE_A, 100G_1X_MODE_A] |
|
|
|
|
|
--link_mode_force |
Configure Link Mode Force (Disable AN) |
|
|
|
-l |--loopback <loopback> |
Configure Loopback Mode [NO(no loopback)/RM(phy remote Rx-to-Tx loopback)/PH(internal phy Tx-to-Rx loopback)/EX(external loopback connector needed)/LL(link layer local loopback)/TRAN(transceiver-level loopback, requires --side and --state)/NEA(near-end analog)/NED(near-end digital)] |
|
|
|
|
|
--side |
Transceiver loopback side selector (only with -l TRAN) [host|media] |
|
|
|
|
--state |
Transceiver loopback state (only with -l TRAN) [input|output|disable] |
|
|
|
-k |--fec <fec_override> |
Configure FEC [AU(Auto)/NF(No-FEC)/FC(FireCode FEC)/RS(RS-FEC)/LL(LL-RS-FEC)/DF-RS(Interleaved_RS-FEC)/DF-LL(Interleaved_LL_RS-FEC)/RS-544Q(Interleaved Quad RS-FEC 544,514)/RS-544(RS-FEC 544,514)/LL-272(LL-RS-FEC 272,257+1)/QUAD_KP4_FEC(Interleaved Quad RS-FEC 546,516 + PLR)/OCTET_KP4_FEC(Interleaved Octet RS-FEC 546,516 + PLR)/Int_KP4_FEC_PLR(Interleaved RS-FEC 544,514 + PLR)/PLR(RS-FEC 544,514 + PLR)/PLR(LL-FEC 271,257 + PLR)/DOUBLE_KP4_FEC(RS-FEC 546,516 + PLR Double)/QUAD_KP4_FEC(RS-FEC 546,516 + PLR Quad)/OCTET_KP4_FEC(RS-FEC 546,516 + PLR Octet)/DOUBLE_HK4_FEC(RS-FEC 288,258 + PLR Double)/QUAD_HK4_FEC(RS-FEC 288,258 + PLR Quad)/OCTET_HK4_FEC(RS-FEC 288,258 + PLR Octet)/DOUBLE_KP4_NP(RS-FEC 546,516 Double, no PLR)] |
|
|
|
|
|
--fec_speed <fec_speed> |
Speed to Configure FEC [100G, 56G, 50G, 40G, 25G, 10G, 1600G_8X, 800G_4X, 400G_2X, 200G_1X, 800G_8X, 400G_4x, 400G_8X, 200G_2X, 200G_4X, 100G_2X, 50G_1X, 100G_4X, 100G_1X_MODE_A, 169G_1X_MODE_A, 176G_1X_MODE_A, 187G_1X_MODE_A] (Default is Active Speed) |
|
|
|
--serdes_tx <params> |
Configure Transmitter Parameters [polarity,ob_tap0,...] |
|
|
|
|
|
--serdes_tx_lane <transmitter_lane> |
Transmitter Lane to Set (Optional - Default All Lanes) |
|
|
|
|
--database |
Save Transmitter Configuration for Current Speed Permanently (Optional) |
|
|
|
|
--tx_params_override |
Set the parameters according to Data Base only, otherwise it will be set according to the best possible configuration chosen by the system (e.g. KR-startup) (Optional) |
|
|
|
--tx_group_map <group_num> |
Map ports to group <group_num> (for NVIDIA Spectrum-2 and NVIDIA Quantum devices) |
|
|
|
|
|
--ports <ports> |
Ports to be mapped [1,2,3,4..] |
|
|
|
--test_mode <prbs_mode> |
Physical Test Mode Configuration [EN(enable)/DS(disable)/TU(perform tuning)] |
|
|
|
|
|
--rx_prbs <rx_prbs_mode> |
RX PRBS Mode [PRBS31, PRBS23A, PRBS23B, PRBS23C, PRBS23D, PRBS7, PRBS11, PRBS11A, PRBS11B, PRBS11C, PRBS11D, PRBS9, IDLE, SQUARE_WAVEA, SQUARE_WAVEB, SQUARE_WAVEC, SQUARE_WAVED, PRBS13A, PRBS13B, PRBS13C, PRBS13D, SSPR, SSPRQ, LT_frames, PRBS15, PRBS28] (Optional - Default PRBS31) |
|
|
|
|
--tx_prbs <tx_prbs_mode> |
TX PRBS Mode [PRBS31, PRBS23A, PRBS23B, PRBS23C, PRBS23D, PRBS7, PRBS11, PRBS11A, PRBS11B, PRBS11C, PRBS11D, PRBS9, IDLE, SQUARE_WAVEA, SQUARE_WAVEB, SQUARE_WAVEC, SQUARE_WAVED, PRBS13A, PRBS13B, PRBS13C, PRBS13D, SSPR, SSPRQ, LT_frames, PRBS15, PRBS28, SQUARE_WAVE3, SQUARE_WAVE13, SQUARE_WAVE30] (Optional - Default PRBS31) |
|
|
|
|
--rx_rate <rx_lane_rate> |
RX Lane Rate [XDR, NDR, HDR, EDR, FDR10, FDR, QDR, DDR, SDR, 1600G_8X, 800G_4X, 400G_2X, 200G_1X, 800G_8X, 400G_4X, 200G_2X, 100G_1X, 400G_8X, 200G_4X, 100G_2X, 50G_1X, 100G, 100G_4X, 50G, 50G_2X, 25G, 40G, 10G, 2.5G, 1G, 400G_2X_MODE_B, 360G_2X_MODE_B, 328G_2X_MODE_B, 378G_2X_MODE_B, 345G_2X_MODE_B, 187G_1X_MODE_A, 176G_1X_MODE_A, 169G_1X_MODE_A] (Optional - Default 25G) |
|
|
|
|
--tx_rate <tx_lane_rate> |
TX Lane Rate [XDR, NDR, HDR, EDR, FDR10, FDR, QDR, DDR, SDR, 1600G_8X, 800G_4X, 400G_2X, 200G_1X, 800G_8X, 400G_4X, 200G_2X, 100G_1X, 400G_8X, 200G_4X, 100G_2X, 50G_1X, 100G, 100G_4X, 50G, 50G_2X, 25G, 40G, 10G, 2.5G, 1G, 400G_2X_MODE_B, 360G_2X_MODE_B, 328G_2X_MODE_B, 378G_2X_MODE_B, 345G_2X_MODE_B, 187G_1X_MODE_A, 176G_1X_MODE_A, 169G_1X_MODE_A] (Optional - Default 25G) |
|
|
|
|
--invert_tx_polarity |
PRBS TX polarity inversion (Optional - Default No Inversion) |
|
|
|
|
--invert_rx_polarity |
PRBS RX polarity inversion (Optional - Default No Inversion) |
|
|
|
|
--force_tx_allowed |
Force TX allowed for PRBS test mode |
|
|
|
|
--skip_power_good_check |
Skip power good check for PRBS test mode |
|
|
|
|
--sysfs_path <path> |
Full path to sysfs files (for SW controlled module) |
|
|
|
|
--lanes |
PRBS lanes to set (one or more lane separated by comma)[0,1,2,...] Optional: Default all lanes |
|
|
|
|
--rx_modulation |
RX Modulation [NRZ/PAM4/PAM4_PRECODING/PAM4_NO_GRAY] (Optional - Default based on lane rate) |
|
|
|
|
--tx_modulation |
TX Modulation [NRZ/PAM4/PAM4_PRECODING/PAM4_NO_GRAY] (Optional - Default based on lane rate) |
|
|
|
|
--primary |
Set the current device as primary for PRBS test mode. Valid only with NVLink6 rates. If omitted on those rates, the device is secondary. |
|
|
|
|
--secondary |
Set the current device as secondary for PRBS test mode. Valid only with NVLink6 rates. This is the default when neither flag is given. |
|
|
|
|
--dc_cpl_allow |
Required to enter test mode on a DC-coupled port. Rejected on a port that is not DC-coupled. Warns that both sides of the link must be powered on before test mode is enabled. |
|
|
|
-b |--ber_collect <csv_file> |
Port Extended Information Collection [CSV File] |
|
|
|
|
--amber_collect <csv_file> |
AmBER Port Extended Information Collection For 16nm Products and Later [CSV File] Note: For --port_type PCIE, a device with more than one PCIe link requires --depth, --pcie_index, and --node. A single PCIe link is selected automatically. |
|
|
|
|
|
--amber_index <page> |
AmBER group selection, working with amber_collect option [1(GENERAL)/2(INDEXES)/3(Link_STATUS)/4(MODULE_STATUS)/5(SYSTEM)/6(HDR_SERDES)/7(NDR_SERDES)/8(PORT_COUNTERS)/9(TROUBLESHOOTING)/10(PHY_OP)/11(LINK_UP)/12(LINK_DOWN)/13(TEST_MODE)/14(MODULE_TEST_MODE)/16(EXT_MODULE_STATUS)/17(SERDES_5NM_GEN7)/20(RECOVERY_COUNTERS)/21(SERDES_5NM_GEN8)] |
|
|
|
|
--amber_rewrite |
Rewrite the csv file instead of appending to it, working with amber_collect option (required when the file columns do not match the selected amBER sheets) |
|
|
|
--ber_limit <limit_criteria> |
BER Limit Criteria [Nominal(Default)/Corner/Drift] (Optional - Default Nominal) |
|
|
|
|
|
--iteration <iteration> |
Iteration Number of BER Collection |
|
|
|
--pc |
Clear Counters |
|
|
|
|
--set_external_phy |
Set External PHY Note: The flag is supported in NVIDIA Spectrum switch systems only. |
|
|
|
|
|
--twisted_pair_force_mode <twisted_pair_force_mode> |
Twisted Pair Force Mode [MA(Master)/SL(Slave)] |
|
|
|
--cable |
Perform operations on the cables |
|
|
|
|
|
--dump |
Dump cable pages in raw format |
|
|
|
|
--ddm |
Get cable Digital Diagnostic Monitoring information |
|
|
|
|
--read |
Perform read operation from specific page |
|
|
|
--length |
Length of data to read in bytes (Optional - Default 1 byte) |
|
|
|
|
--page |
Specific page number to read/write |
|
|
|
|
--offset |
Specific page offset to read/write |
|
|
|
|
|
--write <bytes> |
Perform write operation with specific data (list of bytes, separated by ',') |
|
|
|
|
--prbs_select <side> |
Module PRBS test mode side selector [MEDIA, HOST] |
|
|
|
--prbs_mode |
Perform PRBS test mode on the Module [EN(Enable),DS(Disable)] |
|
|
|
|
--generator_pattern |
Set PRBS generator pattern [PRBS31(default),PRBS23,PRBS7,PRBS11,PRBS9,PRBS13,SSPR,SSPRQ] |
|
|
|
|
--swap_generator |
Enable PAM4 MSB <-> LSB generator swapping (Optional) |
|
|
|
|
--invert_generator |
Enable PRBS generator inversion (Optional) |
|
|
|
|
--generator_lanes |
PRBS generator lanes to set (one or more lane separated by comma)[0,1,2,3,4,5,6,7] (Optional - Default all lanes) |
|
|
|
|
--checker_pattern |
Set PRBS checker pattern [PRBS31(default),PRBS23,PRBS7,PRBS11,PRBS9,PRBS13,SSPR,SSPRQ] |
|
|
|
|
--swap_checker |
Enable PAM4 MSB <-> LSB checker swapping (Optional) |
|
|
|
|
--invert_checker |
Enable PRBS checker inversion (Optional) |
|
|
|
|
--checker_lanes |
PRBS checker lanes to set (one or more lane separated by comma)[0,1,2,3,4,5,6,7] (Optional - Default all lanes) |
|
|
|
|
--lane_rate |
Set PRBS checker and generator lane rate [HDR(50G)(default),1.25G,SDR(2.5G),10.3125G,FDR(14G),EDR(25G),NDR(100G)] |
|
|
|
|
--show_diagnostic_info |
Show PRBS diagnostic counters information |
|
|
|
|
--clear_diagnostic_info |
Clear PRBS diagnostic counters |
|
|
|
|
|
--control_parameters |
Show Module Control Parameters |
|
|
|
--tx_equalization |
Set Module Tx Input Equalization in dB [NE(No Equalization),1,2,3,4,5,6,7,8, 9,10,11,12] |
|
|
|
|
--rx_emphasis |
Set Module RX Output Emphasis in dB. for CMIS, pre-emphasis value will be set [NE(No Equalization),0.5,1,1.5,2,2.5,3,3.5,4,5,6,7] |
|
|
|
|
--rx_post_emphasis |
Set Module Rx Post Emphasis in dB [NE(No Equalization),1,2,3,4,5,6,7] |
|
|
|
|
--rx_amplitude |
Set Module Rx Output Amplitude [0(100-400mV),1(300-600mV),2(400-800mV),3(600-1200mV)] |
|
|
|
|
|
--show_module_cap |
Show module capabilities: host and line precoding support, and transceiver-level loopback capability. Used with --cable. |
|
|
|
--margin |
Read the SerDes eye margins per lane |
|
|
|
|
|
--measure_time <time>
|
Measure time in seconds for single eye [10, 30, 60, 90, 120, 240, 480, 600 and 900] (Optional - Default 60 for PCIe and 30 for Network ports) |
|
|
|
|
--eye_select <eye_sel> |
Eye selection for PAM4 [UP, MID, DOWN, ALL] (Default ALL) |
|
|
|
|
--lane <lane_index> |
Run eye for specific lane index (Default all lanes) |
|
|
|
--rx_error_injection |
Enable the RX link deterioration |
|
|
|
|
|
--mixer_offset0 <value> |
Fine change to the center of the eye [0x0 to 0x7ff] |
|
|
|
|
--mixer_offset1 <value> |
Coarse change to the center of the eye [0x0 to 0x3ff] |
|
|
|
|
--show_mixers_offset |
Show mixer offset 0 and mixer offset 1 |
|
|
|
--rx_fec_histogram |
Provide histogram of FEC errors. The result is divided to bins. Each bin is holding different number of errored bit within FEC protected block |
|
|
|
|
|
--show_histogram |
Show FEC errors histogram |
|
|
|
|
--clear_histogram |
Clears FEC errors histograms |
|
|
|
--pcie_error_injection |
Start/show PCIe error injection (for HCA only) |
|
|
|
|
|
--error_type <type> |
PCIe error type [ABORT(0),BAD_DLLP_LCRC(1),BAD_TLP_LCRC(2),BAD_TLP_ECRC(3),ERR_MSG(4),MALFORMED_TLP(5),POISONED_TLP(6),UNEXPECTED_CPL(7),ACS_VIOLATION(8),SURPRISE_LINK_DOWN(100),RECEIVER_ERROR(101)] |
|
|
|
|
--error_duration <duration> |
Error duration, depend on the error type |
|
|
|
|
--injection_delay <delay> |
Delay in micro-seconds before applying the error (Optional) |
|
|
|
|
--error_parameters <params> |
Comma-separated parameters for selected error type (param0,param1,param2,param3) |
|
|
|
|
--dbdf <dbdf> |
Destination bus device function, e.g af:00.0 (Optional used for specific error type) |
|
|
|
--yes |
Non-interactive mode, answer yes to all questions |
|
|
|
|
--link_training <link_training_mode> |
Link Training Optimization Configuration EN(enable)/DS(disable)/EN_EXT(enable extra) |
|
|
|
|
--phy_recovery <phy_recovery_mode> |
Configure PHY Recovery EN(enable)/DS(disable) |
|
|
|
|
|
--recovery_type <type> |
Configure Recovery Type host_serdes_feq/host_logic_re_lock/phy_recovery_steps |
|
|
|
|
--recovery_policy |
explicit_config_controls / recovery_off / robust_optimized / gradual_recovery. |
|
|
|
|
--step1_preset |
fw_default / peq_only / cdr_toggle / full_duplex_eq / logic_lock_only / redo_link_training / cpo_recovery / no_request. |
|
|
|
|
--step2_preset |
Same list as --step1_preset. |
|
|
|
|
--recovery_rx_algo_time <0-255> |
UPHY Rx algorithm time in ms. 0 means firmware-handled. |
|
|
|
|
--recovery_tx_toggle_time <0-255> |
UPHY Tx toggle time in ms. 0 means firmware-handled. |
|
|
|
|
--recovery_tx_toggle_delay <0-255> |
UPHY Tx toggle delay in ms. 0 means firmware-handled. |
|
|
|
|
--recovery_tx_re_iter |
fw_default / rx_only / tx_and_rx. |
|
|
|
|
--link_down_timeout <0-65535> |
Milliseconds before the port is announced down. 0 announces immediately. 65535 tries every allowed recovery type first. |
|
|
|
|
--link_down_timeout_default |
Restore the link-down timeout to its default. |
|
|
|
--set_link_peq <interval> |
Set link Periodic Equalization (PEQ) interval [10-40000] uS. 0 means FW-Default Only multiples of 10 are supported. |
|
|
|
|
--set_tx_precoding |
Set TX Precoding [EN(enable)/DS(disable)]; with --cable operates at module level and requires --side [host|line] |
|
|
|
|
--set_rx_precoding |
Set RX Precoding [EN(enable)/DS(disable)]; with --cable operates at module level and requires --side [host|line] |
|
|
|
|
--set_plr |
Set PLR configuration. Requires at least one of --plr_reject_mode, --plr_margin_threshold, or --plr_tx_crc. The given parameters are applied in a single write. |
|
|
|
|
|
--plr_reject_mode <reject_mode> |
PLR Reject Mode: 0/MARGIN (PLR margin), 1/CRC_CS (CRC and CS), 2/CS (CS only) (Optional). The mode must be one the device supports. |
|
|
|
|
--plr_margin_threshold <margin_th> |
PLR Margin Threshold [0-7] (Optional). The value must be one the device supports. |
|
|
|
|
--plr_tx_crc <tx_crc> |
TX CRC over PLR: 0/DS(disable), 1/EN(enable) (Optional). Enabling it requires device support. |
|
|
|
--set_apsu |
|
|
|
|
|
|
--set_apsu |
Set APSU (ILT/RTS) Configuration. Takes one or more of the parameters below; they are applied in a single write |
|
|
|
|
--apsu_ctl <ctl> |
APSU [EN(enable)/DS(disable)] (Optional) |
|
|
|
|
--apsu_training_en <training_en> |
Training Enable [EN/DS] (Optional) |
|
|
|
|
--apsu_peer_det <peer_det> |
LT Peer Detection [EN/DS] (Optional) |
|
|
|
|
--apsu_hop_count_ctl <hop_count_ctl> |
Hop Count Control [EN/DS] (Optional) |
|
|
|
|
--apsu_nlut <nlut> |
nLUT Control [EN/DS] (Optional) |
|
|
|
|
--apsu_validation_en <validation_en> |
ILT Frame Validation [EN/DS] (Optional) |
|
|
|
|
--apsu_restart |
Restart Training (Optional) |
|
|
|
|
--apsu_recovery_timer <recovery_timer> |
Recovery Timer in ms [0-255] (Optional) |
|
|
|
|
--apsu_max_recovery_events <max_recovery_events> |
Max Recovery Events [0-255] (Optional) |
|
|
|
|
--apsu_max_wait_timer <max_wait_timer> |
Max Wait Timer in seconds [0-65535] (Optional) |
|
|
|
|
--apsu_propagation_timer <propagation_timer> |
Propagation Timer in 10ms units [0-31] (Optional) |
|
|
|
|
--apsu_quiet_timer <quiet_timer> |
Quiet Timer in 10ms units [0-15] (Optional) |
|
|
|
|
--apsu_invert_to <invert_to> |
Invert Timeout in ms [0-15] (Optional) |
|
|
|
|
--apsu_tf_detect_timer <tf_detect_timer> |
TF Detect Timer in ms [0-15] (Optional) |
|
|
|
|
--apsu_peer_det_timer <peer_det_timer> |
Peer Detection Timer in 100ms units [0-15] (Optional) |
|
|
|
|
--apsu_fail_timer <fail_timer> |
Fail Timer in 10ms units [0-255] (Optional) |
|
|
|
|
--apsu_validation_timer <validation_timer> |
Validation Timer in 10ms units [0-15] (Optional) |
|
|
|
|
--apsu_restart_on_los <restart_on_los> |
Restart On LoS: 0(no restart), 1(restart), 2(restart on long LoS only) (Optional) |
|
|
|
|
--apsu_local_pattern_tx <local_pattern_tx> |
Local Pattern TX: 0(PRBS31), 1(PCS data) (Optional) |
|
|
|
|
--apsu_max_hop_count <max_hop_count> |
Max Hop Count [0-3] (Optional) |
|
|
|
|
--apsu_hop_decrement <hop_decrement> |
Hop Decrement [0-3] (Optional) |
|
|
|
|
--apsu_hop_count <hop_count> |
Hop Count [0-3] (Optional) |
|
|
|
|
--apsu_bcnl_pages_retransmissions <bcnl_pages_retransmissions> |
BCNL Pages Retransmissions [1-255] (Optional) |
|
|
|
--els_module <module_index> |
Enable ELS mode and specify the module index to operate on |
|
|
|
|
--els_laser <lasers> |
Specify which lasers to operate on (separated by commas) |
|
|
|
|
--els_operation <operation> |
Specify the ELS operation to perform: FC(fiber_check) / LT(laser_tuning) / WLT(laser_wl_tuning) / UP(laser_up) / [internal]F_UP(force_laser_up) / [internal]OFF(laser_off) |
|
|
|
|
--save_laser_setpoint <power_setpoint> |
CPO ELS: save the laser power setpoint for the lasers in --els_laser. Requires --els_module. Units are 0.01 mW (10 uW), range [0-65535]. Without --cable the value is written through PMMP. With --cable it is written to the module EEPROM, and each selected laser must already have passed fiber check. |
|
|
|
|
--set_primary |
Set this port as the primary port. Optional --constant_role EN keeps the role persistent; --constant_role DS clears it and returns the role to Auto. |
|
|
|
|
--set_secondary |
Set this port as the secondary port. Optional --constant_role EN keeps the role persistent; --constant_role DS clears it and returns the role to Auto. |
|
|
|
|
|
--constant_role <EN/DS> |
Keep the primary/secondary role constant [EN(enable)/DS(disable)]. Must be used with --set_primary or --set_secondary. DS clears the persistent role and returns it to Auto. |
|
|
|
--set_bkv_group <group_id> |
Set BKV Group Masks |
|
|
|
|
|
--lane <lane_index> |
lane index (Required) |
|
|
|
|
--bkv_rates |
BKV Rates separated by comma: [312.5M,53.125G,106.25G,200G,212.5G] (Optional) |
|
|
|
|
--bkv_roles |
BKV Roles [TLM,RLM,TCLM] (Optional) |
|
|
|
|
--bkv_mode_b_roles |
BKV Mode B Roles [PRIMARY,SECONDARY] (Optional) |
|
|
|
--set_bkv_entry <group_id> |
Set BKV Group Entry |
|
|
|
|
|
--bkv_entry <entry_id> |
BKV Entry ID (Required) |
|
|
|
|
--bkv_address <address> |
BKV Entry Address (Optional) |
|
|
|
|
--lane <lane_index> |
lane index (Required) |
|
|
|
|
--wdata <wdata> |
BKV Entry Write Data (Optional) |
|
|
|
|
--wmask <wmask> |
BKV Entry Write Mask (Optional) |
|
|
Output fields
mlxlink -d <device> -p <port_number> prints these sections before any extra query.
Tool Information
-
Firmware Version
-
amBER Version
-
BKV Version, when the device has one
-
MFT Version
Operational Info
-
State
-
Physical state
-
Speed
-
Width
-
FEC
-
Loopback Mode
-
Auto Negotiation
-
Tx Precoding Status
-
Rx Precoding Status
Port Info
-
Label Port
-
Local Port
-
Primary/Secondary, on NVLink Mode B only
CPO Info
CPO devices only.
-
ELS Index
-
ELS Lane Index
-
OE Index
-
OE Lane Index
Supported Info
-
Enabled Link Speed
-
Supported Cable Speed
On Ethernet with an extended protocol capability, both titles gain (Ext.). On a bonus port, Supported Cable Speed is Loopback.
Troubleshooting Info
-
Status Opcode
-
Group Opcode
-
Recommendation
Physical Counters and BER Info
--show_counters on a network port.
-
Time Since Last Clear [Min]
-
Symbol Errors
-
Symbol BER
-
Effective Physical Errors
-
Effective Physical BER
-
Raw Physical Errors Per Lane
-
Raw Physical BER
-
Raw Physical BER Per Lane
-
Link Down Counter
-
Link Error Recovery Counter
Symbol Errors and Symbol BER are the NDR-generation rows. Older devices print Effective Physical Errors in that place instead.
In PRBS test mode the section is Physical Counters and BER Info (PRBS):
-
Time Since Last Clear [Min]
-
PRBS Errors
-
PRBS BER
FEC Capability Info
--show_fec. One row per supported speed, titled FEC Capability <speed>. On Ethernet products older than 16nm the speed label gains bE. The value is the FEC modes that speed can use.
Device Info
--show_device.
-
Part Number
-
Part Name
-
Serial Number
-
Revision
-
FW Version
BER Monitor Info
--show_ber_monitor. Not supported on HCAs.
-
BER Monitor State
-
BER Monitor Type
External PHY Info
--show_external_phy.
-
Twisted Pair Force Mode
PLR Info
--show_plr.
-
PLR Reject Mode
-
PLR Reject Mode Support
-
PLR Reject Mode Operational
-
PLR Margin Threshold Admin
-
PLR Margin Threshold Support
-
PLR Margin Threshold Operational
-
TX CRC PLR
-
TX CRC PLR Support
APSU Info
--show_apsu.
-
RTS Update State
-
Training Status
-
RTS Status
-
ISL Ready
-
RTS Rx All Lanes
-
RTS Tx All Lanes
-
Remote Type
-
Peer Detected
-
Remote Host IUD
-
Uses Recovered Clock
APSU Lanes Info is one row per lane. Columns: TrainCtlState, TrainStatus, RemoteRTS, LocalRTS, RemoteRxRdy, LocalRxRdy, RemoteTFLock, LocalTFLock, RxOK, Failure.
APSU Configuration echoes the same controls the set flags write. A row is N/A when the page or the capability is absent. Where the register has an operational value, it is appended as (Oper: ...). Restart Training is a trigger and is not printed.
Rows: APSU, Training Enable, LT Peer Detection, Hop Count Control, nLUT Control, ILT Frame Validation, Recovery Timer [ms], Max Recovery Events, Max Wait Timer [s], Propagation Timer [10ms], Quiet Timer [10ms], Invert TO [ms], TF Detect Timer [ms], Peer Detection Timer [100ms], Fail Timer [10ms], Validation Timer [10ms], Restart On LoS, Local Pattern TX, Max Hop Count, Hop Decrement, Hop Count, BCNL Pages Retransmissions.
KR Info
--show_kr.
-
Support Non-Standard Training Flow
-
Number of Iterations
-
Time for Iteration
-
BER Target
-
PRBS Type
Host Class Info
--show_host_class.
-
Local Host Class
-
Remote Host Class
Rx Recovery Counters
--show_rx_recovery_counters. Not supported on HCAs.
Always:
-
Operational Recovery Type
-
Operational Recovery Status
-
Oper Time Host Serdes Feq [msec]
-
Total Successful Recovery Events
-
Time in Last Host Logical Recovery [msec]
-
Time in Last Host Serdes Feq Recovery [msec]
-
Time Since Last Recovery [msec]
-
Last Host Logical Recovery Attempts Count
-
Last Host Serdes Feq Attempts Count
-
Time Between Last 2 Recoveries [msec]. 65535 prints as More than 1 min
NVLink (Mode A or Mode B) also prints:
-
Total RS FEC Uncorrectable During Recovery
-
Last Successful Recovery Step Attempts
-
Total Successful Recovery Time [msec]
-
Last Successful Recovery Time [msec]
NVLink7 also prints:
-
Recovery Policy
-
Link Down Timeout Oper [msec]
-
UPHY Rx Algo Time [msec]
-
UPHY Tx Toggle Time [msec]
-
UPHY Tx Toggle Delay [msec]
-
UPHY Tx Re-Iteration
-
Step1 Preset
-
Step2 Preset
-
Step1 Supported Presets
-
Step2 Supported Presets
-
Time in Last Step1 [msec]
-
Time in Last Step2 [msec]
-
Total Time in Step1 [msec]
-
Total Time in Step2 [msec]
-
Total Step1 Count
-
Total Step2 Count
-
Recovery Entry Reason
-
Last Recovery State
BKV Groups Info
--show_bkv_groups columns: Group #, Supported Entries, Filled Indication.
--show_bkv_group prints BKV Group Properties:
-
Rate Mask
-
Role Mask
-
Mode B Role Mask
BKV Group Entries columns: Entry ID, Address, WData, WMask.
PRR Measurement Data
--show_prr prints PRR Measurement Data (lane <n>). One block per measurement type:
-
PRR meas_type <n>:meas_type 0xNN (<count> DWORDs) -
dword_000,dword_001, and so on, as0x########
Destructive: a link toggle is required afterwards. Requires --lane. Not available together with --plane.
Multi-port tables
--show_multi_port_info / --smpi columns: Port, State, Speed, FEC, Cable Type/LEN, TTLU, Time Since Clear/CL, FOM, Net BER, Link Down, F. C. Zero Hist.
--show_multi_port_module_info / --smpmi columns: Label(Pport). On InfiniBand, Label(Pport)IBport. Then Cable S/N, Cable P/N, Cable Len, Cable Type, State, Speed, FEC, Net BER.
--show_multi_port_cpo_info / --smpci columns: Label(Pport). On InfiniBand, Label(Pport)IBport. Then Status, OE idx, ELS idx, Laser idx. CPO only. FNM ports are skipped.
Cable DDM Information
--cable --ddm.
-
Temperature
-
Voltage
-
Channels
-
RX Power
-
TX Power
-
TX Bias
DDM Flags, for Temperature and Voltage, then per channel for RX Power, TX Power, and TX Bias:
-
Alarm high
-
Alarm low
-
Warning high
-
Warning low
DDM Thresholds columns: Temperature, Voltage, RX Power, TX Power, TX Bias.
-
High alarm threshold
-
High warning threshold
-
Low warning threshold
-
Low alarm threshold
Module PRBS Test Mode
-
Status
-
Lock Status [per lane]
-
PRBS Access
-
PRBS Pattern
-
PRBS Lane Rate
-
PRBS Inversion
-
PRBS MSB ↔ LSB Swap
Module PRBS Diagnostic Counters (--show_diagnostic_info):
-
PRBS Traffic (bits) [per lane]
-
PRBS Errors [per lane]
-
PRBS BER [per lane]
-
Measured SNR [per lane]
Test Mode Info
-
RX PRBS Mode
-
TX PRBS Mode
-
RX Lane Rate
-
TX Lane Rate
-
RX Modulation
-
TX Modulation
-
Tuning Status
-
Lock Status
On NVLink also:
-
Nvlink Mode
-
Primary/Secondary
-
Test Mode FSM State
Module Capabilities
--show_module_cap.
-
Host Precoding Capability
-
Line Precoding Capability
-
Transceiver Level Loopback
Histogram of FEC Errors
Header row: Range, Occurrences. One row per bin, labeled Bin <n>. A single value prints as [<n>]. A span prints as [<min>:<max>], then the occurrence count. The bin count comes from the device.
ELS Operation Results
-
Module Index
-
Operation Bandwidth
If polling times out, mlxlink warns that some operations may still be in progress.
Sub Module <n>: Laser Mask.
Laser <global index>:
-
Required Operation
-
Operation Status
Examples:
Get info of <device>, <port_number>:
mlxlink -d <device> -p <port_number>
Get info of <device>, <port_number> and BER Counters:
mlxlink -d <device> -p <port_number> -c
Get info of <device>, <port_number> and Transmitter Parameters:
mlxlink -d <device> -p <port_number> --show_serdes_tx
Configure Port State:
mlxlink -d <device> -p <port_number> --port_state UP
Configure Port Speeds:
mlxlink -d <device> -p <port_number> --speeds 25G,50G,100G
Configure FEC:
mlxlink -d <device> -p <port_number> --fec RS
Configure Port for Physical Test Mode:
mlxlink -d <device> -p <port_number> --test_mode EN (--rx_prbs PRBS31 --rx_rate 25G --tx_prbs PRBS7 --tx_rate 10G --invert_rx_polarity --invert_tx_polarity)
Configure Port for Physical Test Mode over Software Controlled Module:
mlxlink -d <device> -p <port_number> --test_mode EN (--rx_prbs PRBS31 --rx_rate 25G --tx_prbs PRBS7 --tx_rate 10G --invert_rx_polarity --invert_tx_polarity --force_tx_allowed --skip_power_good_check --sysfs_path /sys/module/sx_core/asic<asic_number>/module<module_number>)
During configuration of the device to enter test mode on software controlled module, mlxink will use the default sysfs path given by the supported OS (/sys/module/sx_core/asic#/module#). Users can provide their own base path using --sysfs_path.
If users are certain that the module is operating correctly, software controlled checks done by the tool can be skipped and enter test mode similar to standard firmware controlled modules.
Perform PRBS Tuning:
mlxlink -d <device> -p <port_number> --test_mode TU
RX and TX lane rates for new devices include the PAM4 speeds (50G_1X and 100G_2X).
eg: mlxlink -d <device> --test_mode EN --rx_rate [normal speeds | 50G_1X | 100G_2X] --tx_rate [normal speeds | 50G_1X | 100G_2X]
The PRBS pattern configured in PAM4 rates is PRBSQ.
Start ELS Operation
mlxlink -d <device> --els_module <module_index> --els_laser <lasers> --els_operation FC
Cable operations:
mlxlink -d <device> --cable [Options]
Dump cable EEPROM pages:
mlxlink -d <device> --cable --dump
Get cable DDM information:
mlxlink -d <device> --cable --ddm
Read from cable:
mlxlink -d <device> --cable --read --page <page number> --offset <bytes offset> --length <number of bytes>
Write to cable:
mlxlink -d <device> --cable --write <bytes separated by comma> --page <page number> --offset <bytes offset>
Configure Transmitter Parameters (on lane, to database):
mlxlink -d <device> -p <port_number> --serdes_tx <polar- ity>,<ob_tap0>,<ob_tap1>,<ob_tap2>,<ob_bias>,<ob_preemp_mode>,<ob_reg>,<ob_leva> (--serdes_tx_lane <lane number>) (--database)
Configure Transmitter Parameters for 16nm devices:
mlxlink -d <device> -p <port_number> --serdes_tx <pre_2_tap>,<pre_tap>,<main_tap>,<post_tap>,<ob_m2lp>,<ob_amp>
Configure Transmitter Parameters for 7nm devices, 100G per lane:
mlxlink -d <device> -p <port_number> --serdes_tx <fir_pre3>,<fir_pre2>,<fir_pre1>,<fir_main>,<fir_post1>,<drv_amp>
Configure Transmitter Parameters for 7nm devices, 50G per lane:
mlxlink -d <device> -p <port_number> --serdes_tx <fir_pre2>,<fir_pre1>,<fir_main>,<fir_post1>,<drv_amp>
Configure Transmitter Parameters for 7nm devices at slower lane rates:
mlxlink -d <device> -p <port_number> --serdes_tx <fir_pre1>,<fir_main>,<fir_post1>,<drv_amp>
Configure Transmitter Parameters for 5nm devices:
mlxlink -d <device> -p <port_number> --serdes_tx <fir_pre3>,<fir_pre2>,<fir_pre1>,<fir_main>,<fir_post1>,<fir_post2>
Getting PCIe links info:
mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --show_links
Valid PCIe Links
----------------
Legend : depth ,pcie_index ,node ,port ,bdf ,link_status
Link 1 : 0 ,0 ,0 ,0 ,00:01:00.0 ,UP
Link 2 : 3 ,0 ,0 ,16 ,00:05:00.0 ,DOWN
Link 3 : 0 ,1 ,0 ,8 ,08:01:00.0 ,UP
Link 4 : 3 ,1 ,0 ,4 ,08:05:08.0 ,UP
..
To query information for a specific link, the depth, pcie_index and node for the link must be specified:
mlxlink -d /dev/mst/mt41682_pciconf0 --port_type PCIE --depth 3 --pcie_index 0 --node 1 --show_serdes_tx --show_eye PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 1
Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4)
Link Width Active (Enabled) : 2X (16X)
EYE Opening Info (PCIe)
-----------------------
Physical Grade : 84, 84
Height Eye Opening [mV] : 1194, 1194
Phase Eye Opening [psec] : 84, 84
Serdes Tuning Transmitter Info (PCIe)
-------------------------------------
Serdes TX parameters : Pol ,tap0 ,tap1 ,tap2 ,bias ,preemp_mode ,reg ,leva
Lane 0 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 1 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 2 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 3 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 4 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 5 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 6 : 0 ,21 ,92 ,7 ,15 ,1 ,10 ,9
Lane 7 : 1 ,21 ,92 ,7 ,15 ,1 ,10 ,9
To print the output in JSON format:
mlxlink -d <device> --show_module --json
Show ELS module info on a CPO system:
mlxlink -d <device> -p <port_number> -m --linkx_els
To show ports group map (for NVIDIA Quantum and NVIDIA Spectrum-2):
mlxlink –d<device> --show_tx_group_map 0
To assign ports to a specific group on NVIDIA Quantum and NVIDIA Spectrum-2:
mlxlink –d <device> --tx_group_map 1 –ports 1,2,3,5,4,8,7,8,9,10,11
To show histogram of FEC errors:
mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --show_histogram
To clear histogram:
mlxlink -d /dev/mst/mt4125_pciconf0 --rx_fec_histogram --clear_histogram
Margin Scan Tool
The margin scan tool is used for scanning PCIe or Network ports (in EDR\25G or HDR\PAM4 speeds).
If the margin scan fails with an "eye scan not completed" message, perform a reboot and run the scan again.
To enable the margin scan with a measure time of 10 seconds:
mlxlink –d <device> --port_type PCIE –margin –measure_time 10
To enable the margin scan for Multi-host or Socket Direct systems through:
-
depth, pcie_index and node:
mlxlink –d <device> --port_type PCIE –depth 0 –pcie_index 1 –node 0 –margin –measure_time 30 -
The local port (can be shown by the
–show_linkscommand):mlxlink –d <device> --port_type PCIE –port 1 –margin –measure_time 10
RX Error Injection
Allows modifying the Eye Center capability by changing the mixer_offset0 (fine change) and mixer_offset1 (coarse change) flags for 28nm products to produce RX errors.
Flags Usage
-
To change the mixers values:
mlxlink -d /dev/mst/mt4117_pciconf0 --rx_error_injection --mixer_offset0 0x200 --mixer_offset1 0x305
Modifying mixer_offset0 and mixer_offset1 flags can change the Eye Center and might cause link degradation.
-
To query the mixers values:
mlxlink -d /dev/mst/mt4117_pciconf0 --rx_erro r_injection --show_mixers_offset
Rx-to-Tx Loopback Mode Activation
This capability enables Rx-to-Tx remote loopback mode.
Prerequisite
-
The port should be disabled and configured with Force mode [Disable the Auto-negotiation]
-
Remote loopback is supported for 25G/50G per lane speed
-
If the NIC has 2 ports, both ports should be configured with the same speed
To enable Rx-to-Tx remote loopback mode:
-
Disable the port.
mlxlink -d /dev/mst/mt4123_pciconf0 –port_state DN -
Configure the link mode to Force [Disable the Auto-negotiation].
mlxlink -d /dev/mst/mt4123_pciconf0 --speeds 100G --link_mode_force -
Configure loopback with remote loopback [RM].
mlxlink -d /dev/mst/mt4123_pciconf0 –loopback RM -
Enable the port.
mlxlink -d /dev/mst/mt4123_pciconf0 –port_state UP
To return to the normal link operation:
-
Disable the port.
mlxlink -d /dev/mst/mt4123_pciconf0 –port_state DNPossible common reasons for failing to bring the port down:
-
Multi Host
-
Socket Direct (SD)
-
Wrong configurations (for example KEEP_ETH_LINK_UP_Px is enabled)
-
Port management is enabled
-
-
Clear the loopback configuration using the "NO loopback" option.
mlxlink -d /dev/mst/mt4123_pciconf0 –loopback NO -
Enable the port.
mlxlink -d /dev/mst/mt4123_pciconf0 –port_state UP
Module PRBS Test Mode
The module PRBS test mode can be performed by using the new flags under the --cable command.
Notes
-
This feature supports Active/Optical CMIS modules only
-
Either the media or host side can run with PRBS mode
-
To enable the PRBS test mode, the module should be plugged in and active
Enabling\Disabling The Module PRBS Test Mode
To enable the module PRBS test mode, the side of the module should be selected using the --prbs_select flag. After providing the --cable flag, either the HOST or the MEDIA side should be selected, so the --prbs_mode<EN\DS> can be used to enable or disable the PRBS test mode process.
For example, the following command will enable the PRBS test mode on the HOST side of the module:
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode EN
The command above will put the HOST side of the module in PRBS test mode with default Checker and Generator parameters.
The Checker and Generator parameters can be overridden while enabling the PRBS test mode according to their related flags in the help menu:
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode EN --checker_pattern PRBS13 --invert_checker --generator_pattern PRBS31 --swap_generator --lane_rate HDR
To disable the PRBS test mode, the following command can be executed:
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --prbs_mode DS
Getting Module Information
Some of the module information can be presented using the --show_module flag.
For example, the following command will present the module information for the module connected to port 1 on the switch:
mlxlink -d /dev/mst/mt53120_pci_cr0 --port 1 --show_module
Module Info
-----------
Identifier : OSFP
Compliance : IB HDR,200GBASE-SR4
Cable Technology : 850 nm VCSEL
Cable Type : Optical Module (separated)
OUI : Nvidia
Vendor Name : NVIDIA
Vendor Part Number : MMA4U00-HS
Vendor Serial Number : MT2134FT06348
Rev : A3
Wavelength [nm] : 850
Transfer Distance [m] : 10
Attenuation (5g,7g,12g,25g,53g)[dB]: N/A
FW Version : 40.55.0
Digital Diagnostic Monitoring : Yes
Power Class : 12.0 W max
CDR RX : ON,ON,ON,ON,ON,ON,ON,ON
CDR TX : ON,ON,ON,ON,ON,ON,ON,ON
LOS Alarm : N/A
Temperature [C] : 36 [-10..80]
Voltage [mV] : 3280 [3100..3500]
Bias Current [mA] : 7.460,7.490,7.292,7.348,7.430,7.200,7.392,7.300 [5.492..8.5]
Rx Power Current [dBm] : -40,-40,-40,-40,-40,-40,-40,-40 [-13.372..5.4]
Tx Power Current [dBm] : 0.473,0.477,0.282,0.434,0.766,0.645,0.500,0.500 [-11.427..5.4]
Intra-ASIC Latency [ns] : N/A
Module Datapath Latency [ns] : N/A
Round Trip Latency [ns] : N/A
SNR Media Lanes [dB] : N/A
SNR Host Lanes [dB] : N/A
IB Cable Width : N/A
Memory Map Revision : 64
Linear Direct Drive : 0
Cable Breakout : Unspecified
SMF Length : N/A
MAX Power : 48
Cable Rx AMP : 1
Cable Rx Emphasis (Pre) : 0
Cable Rx Post Emphasis : 0
Cable Tx Equalization : 0
Wavelength Tolerance : 15.0nm
Module State : Ready state
DataPath state [per lane] : DPActivated,DPActivated,DPActivated,DPActivated,DPActivated,DPActivated,DPActivated,DPActivated
Rx Output Valid [per lane] : 0,0,0,0,0,0,0,0
Nominal bit rate : N/A
Rx Power Type : Average power
Manufacturing Date : 24_08_21
Active Set Host Compliance Code : IB HDR
Active Set Media Compliance Code : 200GBASE-SR4
Error Code Response : ConfigUndefined
Module FW Fault : 0
DataPath FW Fault : 0
Tx Fault [per lane] : 0,0,0,0,0,0,0,0
Tx LOS [per lane] : 0,0,0,0,0,0,0,0
Tx CDR LOL [per lane] : 0,0,0,0,0,0,0,0
Rx LOS [per lane] : 0,0,0,0,0,0,0,0
Rx CDR LOL [per lane] : 1,1,1,1,1,1,1,1
Tx Adaptive EQ Fault [per lane] : 0,0,0,0,0,0,0,0
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --clear_diagnostic_info
PRBS Diagnostic Counters Information
After performing the PRBS test mode, the module counters can be queried by using the following command:
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --show_diagnostic_info
The module PRBS test mode counters can be cleared by using the following command, which will clear the diagnostic counters on the HOST side only:
mlxlink -d /dev/mst/mt53104_pciconf0 --port 3 --cable --prbs_select HOST --clear_diagnostic_info
Module Control Parameters
Some of the module parameters can be controlled by mlxlink after providing the --control_paramenter flag, which can be executed under the --cable flag.
The possible parameters can be controlled as follows:
-
Reading and configuring Tx Equalization
-
Reading and configuring Rx Emphasis (PreCursor & PostCursor)
-
Reading and configuring Rx Amplitude
Notes
-
To apply the changes, the link should be disabled first
-
After configuring a new parameter, the link should be raised again to allow the firmware to load the new configuration
-
Cable control parameters are valid for active\optical modules only
Querying and Configuring The Module Control Parameters
To query the currently configured module control parameters, the --control_parameters flag can be used under the --cable flag as follows:
mlxlink -d /dev/mst/mt53104_pciconf0 --cable --control_parameters
...
Module Control Parameters
-------------------------
TX Equalization : 1dB
RX Emphasis (pre) : 2.5dB
RX Post Emphasis : No Equalization
RX Amplitude : 600-1200 mV (P-P)
To configure the module control parameters, the following command can be executed:
mlxlink -d /dev/mst/mt53104_pciconf0 --cable --control_parameters --tx_equalization 2 --rx_amplitude 1
APSU (IEEE 802.3dj ILT/RTS) on NVIDIA ConnectX-9
Beta Level: This feature is under active development and may be unstable. We recommend using it with caution.
APSU (Autonomous Path Startup) is an IEEE 802.3dj mechanism for bringing up high-speed Ethernet PHY paths in a coordinated way, using ILT to train individual inter-sublayer links and RTS to signal readiness across the full path before normal data transmission begins. mlxlink utility shows and configures APSU (ILT/RTS).
To show APSU info (add --json for JSON output):
mlxlink -d <device> -p <port_number> --show_apsu
To set APSU configuration (one or more apsu_* parameters, applied in a single write):
mlxlink -d <device> -p <port_number> --set_apsu --apsu_ctl EN --apsu_training_en EN --apsu_bcnl_pages_retransmissions 95
LTX Status and Logger History on NVLink Ports
mlxlink reports the firmware's LTX record for an NVLink port — the link-training flow whose iterations are accepted or rejected against a FEC/BER measurement. Use it to see whether link training converged on this port, how many retries it took, which threshold failed when it did not, and what the previous training attempts looked like.
The flag is a query, not a command: it is appended to mlxlink's normal port output rather than replacing it, and it can be combined with other queries.
mlxlink -d <device> -p <port_number> --show_ltx
Add --json to get the same data in JSON format.
Note: -p/--port is optional; when it is omitted mlxlink operates on label port 1.
Note: The LTX record is scoped to the current link bring-up. Toggling the port — with --port_state DN/UP or TG — discards it. Capture --show_ltx before touching a port you are investigating.
Supported configurations
|
|
|
|---|---|
|
Ports |
NVLink ports only — GPU and NVSwitch. Support is a property of the port's active protocol, not of the device: a port on an NVIDIA Quantum-3 switch running InfiniBand is rejected. |
|
NVLink generations |
NVL5 and NVL6. The two generations expose this data differently, so mlxlink prints a different set of tables for each. |
|
Port types |
NETWORK only. --port_type PCIE is rejected. |
|
Lanes |
Exactly two lanes (0 and 1) are reported. |
The output shape is chosen automatically from the port's NVLink generation. No flag selects it.
|
Generation |
Sections printed |
|---|---|
|
NVL6 |
LTX Status, LTX Logger History |
|
NVL5 |
LTX FEC Measure (NVL5), LTX FEC Measure (NVL5) Lanes, LTX FEC Measure (NVL5) Iterations |
NVL6 output
LTX Status is the current per-lane verdict; LTX Logger History is the firmware's ring buffer of the last training attempts.
LTX Status
----------
|---------------------------------------------------------------------------------------------------|
| Lane | Status | Fail Reason | Retry | Raw BER |Raw BER Target | Hist | Hist Target |
|---------------------------------------------------------------------------------------------------|
| 0 | FAIL | RAW | 8 | 1.5E-6 | 1.0E-5 | 6 | 9 |
| 1 | PASS | | 0 | 1.5E-6 | 1.0E-5 | 6 | 9 |
LTX Logger History
------------------
|-----------------------------------------------------------------------------------------------------------------------|
| Lane | # | Status | Fail Reason | Retry | Retry Fail | Eff Err | Hist | Raw BER | PRBS BER mag |
|-----------------------------------------------------------------------------------------------------------------------|
| 0 | 0 | FAIL | HISTOGRAM_RAW | 0 | 2 | 0 | 8 | 1.0E-5 | 0 |
| 0 | 1 | FAIL | RAW | 1 | 1 | 0 | 7 | 1.2E-5 | 0 |
| 0 | 2 | PASS | | 2 | 0 | 0 | 6 | 1.5E-6 | 0 |
|-----------------------------------------------------------------------------------------------------------------------|
| 1 | 0 | PASS | | 0 | 0 | 0 | 6 | 1.5E-6 | 0 |
-
Lane: lane index, 0 or 1.
-
Status:
PASSorFAIL. Printed in green forPASSand red forFAIL. A value the tool does not recognize is printed asUNKNOWN_<value>rather than being hidden. -
Fail Reason: which threshold the lane failed. The cell is empty when the lane did not fail. It is a combination of up to four causes — see the decode table below.
-
Retry: how many times the link quality was re-checked by FEC measure before the link came up.
0means the link was good on the first check;1means it needed a second check. -
Raw BER and Hist: the measured raw BER and the highest non-zero histogram bin taken from the most recent passing entry in that lane's logger history — not from the current-state registers. If the lane has never passed, both columns show
-. -
Raw BER Target and Hist Target: the thresholds the measurement is judged against. These do come from the current per-lane registers, so the table reads as measured-versus-target side by side.
BER values are printed as <mantissa>.<fraction>E-<exponent> — for example 1.5E-6.
Fail Reason decode. Read it as a set of flags, not as an ordinal code: each bit says which measurement missed its target, which is why the names compose. A RAW bit means the measured Raw BER missed Raw BER Target; a HISTOGRAM bit means Hist missed Hist Target. The reason is a four-bit combination: EFFECTIVE (effective errors), HISTOGRAM, RAW (raw BER) and PRBS. All sixteen combinations have names:
|
Value |
Name |
Value |
Name |
|---|---|---|---|
|
0 |
(empty in the table; NONE in JSON) |
8 |
PRBS |
|
1 |
EFFECTIVE |
9 |
EFFECTIVE_PRBS |
|
2 |
HISTOGRAM |
10 |
HISTOGRAM_PRBS |
|
3 |
EFFECTIVE_HISTOGRAM |
11 |
EFFECTIVE_HISTOGRAM_PRBS |
|
4 |
RAW |
12 |
RAW_PRBS |
|
5 |
EFFECTIVE_RAW |
13 |
EFFECTIVE_RAW_PRBS |
|
6 |
HISTOGRAM_RAW |
14 |
HISTOGRAM_RAW_PRBS |
|
7 |
EFFECTIVE_HISTOGRAM_RAW |
15 |
EFFECTIVE_HISTOGRAM_RAW_PRBS |
An unrecognized value prints as RESERVED_<value>.
-
#: position in the history, oldest first.
#0is the oldest retained attempt and the last row of a lane is the most recent. This is a chronological index, not the hardware ring slot. -
Retry Fail: how many FEC-measure checks actually failed before the link came up. It counts failed checks, not attempted ones.
-
Eff Err: effective errors counted during the attempt.
-
Hist: highest non-zero histogram bin for the attempt.
-
Raw BER: raw BER measured during the attempt.
-
PRBS BER mag: the PRBS BER magnitude (exponent) only — an integer, not a formatted BER.
The ring holds up to ten attempts per lane. A lane whose ring is empty contributes no rows, and a partially filled ring contributes only the attempts actually recorded. A horizontal rule separates lane 0's block from lane 1's.
NVL5 output
Three tables: one for the port, one per lane, and one per stored BER measurement.
LTX FEC Measure (NVL5)
----------------------
|-----------------------------------------------------------------------------------------------------------------------------------------------|
| Restore Count | Total Rounds | BER Meas Done |Auto Reversals | Limiter Allow | Reached Max Retry | Entered LTX Flow |BER Based In Progress|
|-----------------------------------------------------------------------------------------------------------------------------------------------|
| 3 | 12 | 4 | 0 | 1 | 0 | 1 | 0 |
-
Restore Count: LTX restore retries for this bring-up.
-
Total Rounds: completed LTX cycles.
-
BER Meas Done: number of BER measurements completed.
-
Auto Reversals: automatic reversals applied.
-
Limiter Allow, Reached Max Retry, Entered LTX Flow, BER Based In Progress: 0/1 state flags.
Entered LTX Flowis 1 once the flow has been entered;BER Based In Progressis 1 while a BER-based round is running.
LTX FEC Measure (NVL5) Lanes
----------------------------
|---------------------------------------------------------------------------------------------------|
| Lane | Stores | Winner |Violation | Viol idx | Viol type | Fail stage | Force |
|---------------------------------------------------------------------------------------------------|
| 0 | 2 | 0 | 0 | 0 | | ok | 0 |
| 1 | 1 | 0 | 1 | 0 | no_phy_stamp |phy_update_rejected | 0 |
-
Stores: how many BER measurements this lane stored. This also sets how many rows the lane contributes to the Iterations table.
-
Winner: index of the winning iteration.
-
Violation / Viol idx / Viol type: a handshake-stamp integrity check inside the LTX store flow, not a link-protocol error — whether a stamp mismatch was detected, at which entry, and which stamp was missing. Viol type is empty when there is no violation. Values:
none,no_serdes_stamp(the SerDes never marked the snapshot stored),no_phy_stamp(the PHY never wrote the BER fields),both; anything else prints asunknown_<value>. -
Fail stage: last failure point in the store/force flow —
ok,phy_update_rejected,force_no_valid_entry,phy_bad_index_overflow. Values 1 and 4 are reserved and print asreserved_1andreserved_4; an unrecognized value prints asunknown_<value>. -
Force: whether a forced entry was applied.
LTX FEC Measure (NVL5) Iterations
---------------------------------
|---------------------------------------------------------------------------------|
| Lane | Iter | BER | serdes_valid | phy_valid | meas_invalid |
|---------------------------------------------------------------------------------|
| 0 | 0 | 1.2E-5* | 1 | 1 | 0 |
| 0 | 1 | 2.0E-5 | 1 | 1 | 0 |
| 1 | 0 | 3.0E-5* | 1 | 0 | 1 |
-
Iter: iteration index within the lane. Up to five iterations are stored per lane; a lane that stored nothing contributes no rows.
-
BER: the BER measured for that iteration. A trailing asterisk marks the winning iteration — it is the only marking of the winner in this table.
-
serdes_valid / phy_valid: whether the SerDes and PHY stamps for that measurement are valid.
-
meas_invalid: set when the firmware itself marks the measurement as one to ignore — no PCS lock when the measurement started, or alignment lost part-way through. mlxlink still displays such iterations and still marks the winner among them; it does not filter them out.
Errors
|
Condition |
Message |
|---|---|
|
Used with --port_type PCIE |
"--show_ltx" option is not supported for PCIE |
|
Port's active protocol is not NVLink |
"--show_ltx" is supported on GPU/switch NVLink ports only |
|
The port does not expose the LTX data |
LTX is not supported for the current device! |
All three are fatal: mlxlink prints -E- <message> on stderr and exits with status 1.
Note: When --show_ltx fails, --json produces no JSON output at all — only the -E- line and the exit code. Check the exit code before treating an empty result as "no LTX data".
Note: A comma-separated port list passed to -p/--port cannot be combined with --show_ltx. A port list is currently supported only with --port_state (-a).
Note: An amBER collection (--amber_collect) does not capture LTX data. When attaching evidence to a bug, capture it separately with mlxlink -d <device> -p <port_number> --show_ltx --json.
Example
# mlxlink -d <device> -p <port_number> --show_ltx
LTX Status
----------
|---------------------------------------------------------------------------------------------------|
| Lane | Status | Fail Reason | Retry | Raw BER |Raw BER Target | Hist | Hist Target |
|---------------------------------------------------------------------------------------------------|
| 0 | FAIL | RAW | 8 | 1.5E-6 | 1.0E-5 | 6 | 9 |
| 1 | PASS | | 0 | 1.5E-6 | 1.0E-5 | 6 | 9 |
LTX Logger History
------------------
|-----------------------------------------------------------------------------------------------------------------------|
| Lane | # | Status | Fail Reason | Retry | Retry Fail | Eff Err | Hist | Raw BER | PRBS BER mag |
|-----------------------------------------------------------------------------------------------------------------------|
| 0 | 0 | FAIL | HISTOGRAM_RAW | 0 | 2 | 0 | 8 | 1.0E-5 | 0 |
| 0 | 1 | FAIL | RAW | 1 | 1 | 0 | 7 | 1.2E-5 | 0 |
| 0 | 2 | PASS | | 2 | 0 | 0 | 6 | 1.5E-6 | 0 |
|-----------------------------------------------------------------------------------------------------------------------|
| 1 | 0 | PASS | | 0 | 0 | 0 | 6 | 1.5E-6 | 0 |
Reading this example: lane 1 trained cleanly on its first attempt. Lane 0 needed three attempts — the first failed on both the histogram and the raw BER, the second on the raw BER alone, and the third passed at 1.5E-6 against a 1.0E-5 target. The FAIL and RAW still shown in LTX Status for lane 0 are its current status registers, while the Raw BER and Hist columns report the measurement from the attempt that did pass.
PHY Information (Per-Lane RX Signal Detect and AM Lock)
mlxlink reports the per-lane receive state of the port's own PCS/PHY: whether each lane sees an incoming signal and whether each lane has acquired alignment-marker (AM) lock. It also reports the AM-lock state latched by hardware at the last link-down, which is what makes the command useful for investigating link flaps — the latched row survives the event that the live rows no longer show.
mlxlink -d <device> -p <port_number> --show_phy_info
Add --json to get the same data in JSON format.
Note: --show_phy_info reports the device's own internal PHY. It is separate from --show_external_phy, which reports the master/slave force mode of an external twisted-pair (BASE-T) PHY on Ethernet switches. The two flags share no data.
Output
PHY Info
--------
RX Signal Detect Per Lane : [per-lane signal detect]
RX AM Lock Per Lane : [per-lane AM lock]
HW Link PHY State : [HW link PHY state]
RX AM Lock Latched Per Lane : [per-lane AM lock at last link-down]
-
RX Signal Detect Per Lane: per lane, whether the receiver detects an incoming signal. Shown only when the hardware marks the value valid; otherwise the whole row reads
N/A. -
RX AM Lock Per Lane: per lane, whether the PCS has acquired alignment-marker lock. This is the live value. The hardware provides no validity bit for it, so the row always shows digits — on a down link those digits may be stale.
-
HW Link PHY State: the hardware link PHY state machine. This row is printed on NVLink7 ports only and is absent everywhere else.
-
RX AM Lock Latched Per Lane: the per-lane AM-lock state as latched at the last link-down — that is, which lanes still had AM lock when the link last went down. Shown only when the hardware marks the snapshot valid; otherwise the row reads
N/A, which is what a port that has not gone down since reset will show.
Per-lane values are printed one digit per lane, lane 0 first, comma-separated: 1,0 means lane 0 has the bit set and lane 1 does not. The number of entries follows the port's active width, so expect one or two entries on an NVLink port and up to eight on Ethernet.
Note: On a port whose link is down, mlxlink cannot determine the lane count and all three per-lane rows read N/A — including RX AM Lock Latched Per Lane. The latched row exists for post-mortem, but it is only readable while the link is up. Capture --show_phy_info while the link is up, or immediately after a flap once the link has come back, rather than on a port that is still down.
Note: The AM-lock rows report the RS-FEC alignment-marker lock bits. On a link running No-FEC or Firecode FEC, the corresponding per-lane lock state is held in different hardware bits, which this command does not display.
Note: Nothing in mlxlink clears the latched snapshot. It is read-only.
Note: Do not use --show_phy_info with --port_type PCIE. The flag is not rejected there, but mlxlink never establishes a lane count for a PCIe link, so the per-lane rows are rendered against a default of eight lanes and the digits are meaningless.
HW Link PHY State values. The state is reported as one of the following; a value with no name is printed as raw hexadecimal, for example 0x53.
|
Group |
What it means |
States |
|---|---|---|
|
Sleep / poll |
No trained partner yet |
sleeping_delay, sleeping_quiet, polling_active, polling_quiet |
|
Administratively off |
remote_down_cmd means the peer brought the link down |
disable, local_down_cmd, remote_down_cmd |
|
Bring-up / training |
Normal transient states during link-up |
config_debounce, config_receiver, config_wait_remote, config_tx_reverse_lanes, config_enhanced, config_test, config_wait_remote_test, config_wait_cfg_enhanced, config_idle, update_retimer, retimer_align |
|
Link up |
The healthy steady states |
linkup, linkup_tx_idle, linkup_tx_empty |
|
Recovery |
The link is retraining after an error |
recover_retrain, recover_wait_remote, recover_idle, uphy_recovery_send_ts1, uphy_recovery_send_pam2, uphy_recovery_send_pam4, uphy_recovery_peq |
|
Test / force |
Set deliberately by test or debug flows, not seen in normal operation |
test, force_send_ts1, force_send_ts2, force_sent_idle, force_send_ts_mlnx, force_send_ts3, force_linkup |
|
Low power |
L1 / quiet entry and wake |
go_to_quiet, quiet_entry, quiet, wake, wake_tx_sleep0, send_announce, tx_hs, wait_for_cdr_lock |
Errors
If the PHY data cannot be read, the entire PHY Info section is omitted — mlxlink's other sections still print normally — and the reason is reported at the end of the output under a red Errors heading:
Showing PHY Info raised the following exception: <reason>
mlxlink exits with status 1.
Note: Under --json the error text is suppressed. A failed --show_phy_info yields JSON with no PHY Info object and no error text; only the exit code signals the failure.
Examples
On a healthy NVLink7 port (NVLink ports report one or two lanes):
# mlxlink -d <device> -p <port_number> --show_phy_info
PHY Info
--------
RX Signal Detect Per Lane : 1,1
RX AM Lock Per Lane : 1,1
HW Link PHY State : linkup
RX AM Lock Latched Per Lane : 1,1
On an eight-lane Ethernet port — the HW Link PHY State row is absent on anything that is not NVLink7:
# mlxlink -d <device> -p <port_number> --show_phy_info
PHY Info
--------
RX Signal Detect Per Lane : 1,1,1,1,1,1,1,1
RX AM Lock Per Lane : 1,1,1,1,1,1,1,1
RX AM Lock Latched Per Lane : 1,1,1,1,1,1,1,1
A degraded NVLink7 port — lane 1 has lost both signal and AM lock, the link is retraining, and no valid link-down snapshot is available:
# mlxlink -d <device> -p <port_number> --show_phy_info
PHY Info
--------
RX Signal Detect Per Lane : 1,0
RX AM Lock Per Lane : 1,0
HW Link PHY State : recover_retrain
RX AM Lock Latched Per Lane : N/A
Reading the third example: lane 1 is the one to investigate — a lane with no signal detect and no AM lock points at the physical path for that lane (cable, connector, module) rather than at the link as a whole.
Related commands
On an NVLink port these read data --show_phy_info does not show, and are worth capturing alongside it:
-
--show_rx_recovery_counters— on NVLink7 it reads the same link-down data and decodes why the link entered recovery. Switch and GPU only. -
The default output's Troubleshooting Info section, which carries the status and group opcodes for the port.
Selecting amBER Sheets (--amber_index, --amber_rewrite)
--amber_collect <csv_file> writes one CSV row per port. By default it collects every amBER sheet. Two flags shape that: --amber_index chooses which sheets to collect, and --amber_rewrite controls whether the CSV is appended to or replaced. Both work only together with --amber_collect.
mlxlink -d <device> -p <port_number> --amber_collect <csv_file> --amber_index <page[,page,...]>
mlxlink -d <device> -p <port_number> --amber_collect <csv_file> --amber_rewrite
--amber_index
Takes a comma-separated list of sheet indexes — for example --amber_index 1,3,12. There is no range syntax: 1-5 is rejected. When the flag is omitted, mlxlink collects the whole catalog and the CSV columns follow ascending index order. When it is given, the columns follow the order you typed.
|
Index |
Sheet |
Index |
Sheet |
|---|---|---|---|
|
1 |
GENERAL |
11 |
LINK_UP |
|
2 |
INDEXES |
12 |
LINK_DOWN |
|
3 |
Link_STATUS |
13 |
TEST_MODE |
|
4 |
MODULE_STATUS |
14 |
MODULE_TEST_MODE |
|
5 |
SYSTEM |
16 |
EXT_MODULE_STATUS |
|
6 |
HDR_SERDES |
17 |
SERDES_5NM_GEN7 |
|
7 |
NDR_SERDES |
20 |
RECOVERY_COUNTERS |
|
8 |
PORT_COUNTERS |
21 |
SERDES_5NM_GEN8 |
|
9 |
TROUBLESHOOTING |
|
|
|
10 |
PHY_OP |
|
|
Note: The list is deliberately not contiguous. Indexes 15, 18 and 19 are not selectable — 15 exists in firmware but is not exposed by the tool, and 18 and 19 are unused. Passing any of them is rejected.
Errors:
|
Condition |
Message |
|---|---|
|
Index not in the table above |
Invalid sheet index [<n>], check the help menu for valid sheets indexes |
|
The same index given twice |
Duplicated sheet index [<n>], please fix the duplication and run again |
|
Leading, trailing, or empty list element ("1," or ",1" or "1,,2") |
Wrong input format |
|
Used without --amber_collect |
amber_index option is used with amber_collect |
--amber_rewrite
Takes no argument. By default --amber_collect appends to the CSV, which is what lets a file be built up across runs. Before appending, mlxlink compares the file's existing header line against the header this run would produce; if they differ it refuses:
The columns of <csv_file> do not match the selected amBER sheets, use --amber_rewrite or provide a different file
That is the situation --amber_rewrite exists for — most often after changing --amber_index and reusing the same file. The flag replaces the file's contents instead of appending to them.
Note: The file is truncated once per run, not once per port. The first row written replaces the file; every further port in the same invocation is appended after it. Collecting several ports with --amber_rewrite therefore yields one file containing all of them.
Used without --amber_collect, the flag fails with:
When using the amber_rewrite option, the amber_collect flag must be provided!
Change in behavior: amBER over PCIe now requires the link to be named
This changes the result of a command that previously succeeded. When collecting amBER over PCIe on a device that exposes more than one PCIe link, the link must now be identified explicitly.
Previously, mlxlink -d <device> --port_type PCIE --amber_collect <csv_file> on such a device wrote a single CSV row for one of the links, chosen arbitrarily, with nothing to indicate that the others had been dropped. The collection now stops instead, and nothing is written.
Name the link in either of two ways — the DPN triple, or the port:
mlxlink -d <device> --port_type PCIE --depth <d> --pcie_index <i> --node <n> --amber_collect <csv_file>
mlxlink -d <device> --port_type PCIE -p <port_number> --amber_collect <csv_file>
Use --show_links to list the device's PCIe links with their depth, pcie_index and node values.
|
PCIe links on the device |
Result |
|---|---|
|
One |
Unchanged — the link is selected automatically and no DPN is needed. |
|
More than one, no link named |
Collection stops with The --depth, --pcie_index and --node must be specified! |
|
More than one, link named |
Collects that link. |
|
None detected |
Collection stops with No valid depth, node and pcie_index detected! |
The check runs only for --port_type PCIE together with --amber_collect, and only on 16nm and later products. It does not affect amBER collection on network ports, and it does not affect -b/--ber_collect.
Note: This is reported as a collection error, not a fatal argument error. The message appears at the end of the output under the red Errors heading, prefixed Collecting AMBER raised the following exception:, and mlxlink exits with status 1. Because the check runs before collection starts, the usual Collecting amBER and producing report to <csv_file> progress line is not printed, and the CSV file is left untouched.
Tool Usage with NIC vs. Switch (-p Flag)
When using the mlxlink tool with an adapter card, notice that the "label_port" -p flag should not be used. To address different ports, please use different MST devices.
For example:
To address port 1 when using ConnectX-4:
mlxlink -d /dev/mst/mt4115_pciconf0
To address port 2, use:
mlxlink -d /dev/mst/mt4115_pciconf0.1
-
Any mlxlink command for a switch should include the "
-p" flag to address the specific port in the switch -
When working with an adapter card, if an MTUSB is used for communication with the NVIDIA NIC, to address port 2, use
mlxlink -d /dev/mst/mt4115_pciconf0 --gvmi_address<0xAddress>
Tool Usage on NVIDIA Quantum HDR Switch Systems with Split Ports
If the split port number is not provided by the ibdiagnet tool, to use mlxlink on NVIDIA Quantum HDR based switch systems split ports, run:
mlxlink -d lid-<LID> -p <formula>
Formula:
In case of 2X port:
-
1- port_num = round_down[(Iblinkinfo_port_num + 1)*0.5]
-
2- if (Iblinkinfo_port_num + 1) modulo 2 =1 then append ‘/2’ to port_num
In case of 4X port, use only item #1 above.
Example:
43 23[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12
43 24[ ] ==( 2X 53.125 Gbps Active/ LinkUp)==> mlxlink -d lid-43 -p 12/2
Tool Usage on NVIDIA Quantum-2 NDR Switch Systems
In NVIDIA Quantum-2 NDR switch systems, there are 32-OSFP cages (8x), where each one holds 2 (4x) ports instead of 1, and each port can be accessed by providing the cage number and the port in the cage - “Cage/Port”.
mlxlink -d <mst deivce> -p <Cage>/<Port>
If the split profile is ready, it is possible to access the split ports by providing the number of split to the port flag, e.g.:
-
To access the main port of 15/2:
mlxlink -d <mst deivce> -p 15/2 -
To access the split port of 15/2:
mlxlink -d <mst deivce> -p 15/2/2
PCIe
Link Speed and Width
For PCIe link speed and width, use the following flag: --port_type PCIE.
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : [Depth, pcie index, node]
Link Speed Active (Enabled) : [Freq – Gen]
Link Width Active (Enabled) : [Width]
PCIe Switch
When using NVIDIA ConnectX-5 and newer devices, the PCIe interface can be configured for a PCIe switch. When the PCIe switch is enabled, the depth, pcie_index and node parameters are needed in order to specify the PCIe port from which the requested information (such as counters or eye info) is gathered.
|
Parameters |
Description |
|---|---|
|
Depth |
This parameter defines the number of layers from the Root Complex to the specific port.
|
|
Pcie_index |
This parameter defines the root complex ID or host index.
|
|
Node |
This parameter defines the specific PCIe port.
Note: For NVIDIA BlueField/BlueField-2 SmartNIC mode, the PCIe link information can only be gathered from the external host. The PCIe interface status cannot be retrieved from the Arm side. When retrieving the PCIe link information from the external host, there is no need to specify the depth, pcie_index and node. |
Example: NVIDIA BlueField JBoF Mode
# mlxlink -d /dev/mst/mt41682_pciconf0 --port_type pcie --depth 3 --pcie_index 0 --node 4 -c
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 4
Link Speed Active (Enabled) : 8G-Gen 3 (16G-Gen 4)
Link Width Active (Enabled) : 2X (2X)
Management PCIe Timers Counters Info
------------------------------------
dl down : 0
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : 0
TX Errors : 0
CRC Error dllp : 0
CRC Error tlp : 0
Link Counters
For PCIe counters information, use the --port_type PCIE –c flag.
Management PCIe Timers Counters Info
------------------------------------
dl down : [link down counter]
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : [Rx Errors]
TX Errors : [Tx Errors]
CRC Error dllp : [CRC Errors dllp]
CRC Error tlp : [CRC Errors tlp]
FEC Correctable Error count : [FEC correctable errors]
FEC Uncorrectable Error count : [FEC uncorrectable errors]
First BER (FBER) : [First BER]
Effective ber : [Effective BER]
Flit Error Ratio : [Flit Error Ratio]
-
RX Errors: indicate the number of transitions to recovery required due to framing errors and CRC (dlp and tlp) errors.
-
TX Errors: indicate the number of transitions to recovery required due to EIEOS and TS errors.
-
CRC Error dllp: indicate CRC error in Data Link Layer Packets.
-
CRC Error tlp: indicate CRC error in Transaction Layer Packet.
-
Flit Error Ratio: indicate the ratio of flits received with errors, presented only when FLIT mode is active.
-
FEC Correctable Error count: presented only when FLIT mode is active.
-
FEC Uncorrectable Error count: presented only when FLIT mode is active.
-
First BER (FBER): 5nm and later, and FLIT mode is active.
-
Effective ber: 16nm and later.
Example:
# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -c
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
Management PCIe Timers Counters Info
------------------------------------
dl down : 3
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : 0
TX Errors : 16
CRC Error dllp : 0
CRC Error tlp : 0
Link Eye Opening and Grade
For PCIe link physical grade and eye opening information, use the --port_type PCIE –e flag.
EYE Opening Info (PCIE)
------------------------
Physical Grade : [Grade0, Grade1, Grade2, Grade3, Grade4, Grade5, Grade6, Grade7, Grade8, Grade9, Grade10, Grade11, Grade12, Grade13, Grade14, Grade15]
Height Eye Opening [mV] : [Height0, Height1, Height2, Height3, Height4, Height5, Height6, Height7, Height8, Height9, Height10, Height11, Height12, Height13, Height14, Height15]
Phase Eye Opening [psec] : [Phase0, Phase1, Phase2, Phase3, Phase4, Phase5, Phase6, Phase7, Phase8, Phase9, Phase10, Phase11, Phase12, Phase13, Phase14, Phase15]
Example:
# mlxlink -d /dev/mst/mt4123_pciconf0 --port_type PCIE -e
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
EYE Opening Info (PCIe)
-----------------------
Physical Grade : 57279, 56340, 59340, 61824, 55140, 60501, 61530, 57392, 61573, 58930, 62752, 60421, 57188, 59796, 60066, 60847
Height Eye Opening [mV] : 292, 288, 314, 325, 278, 310, 319, 299, 316, 318, 343, 323, 310, 311, 335, 318
Phase Eye Opening [psec] : 30, 30, 30, 30, 30, 30, 30, 30, 30, 28, 28, 28, 28, 30, 28, 30
CPU
On an NVIDIA Vera CPU, mlxlink works over PCIe only: --port_type PCIE is required, and the supported queries are the PCIe link information, --show_links, --show_counters and --show_eye. Any other query or operation is rejected with an error.
Example: PCIe link information
# mlxlink -d /dev/mst/mt12181_pciconf0 --port_type PCIE
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 32G PAM-4 Gen6 (32G PAM-4 Gen6)
Link Width Active (Enabled) : 16X (16X)
Example: valid PCIe links
# mlxlink -d /dev/mst/mt12181_pciconf0 --port_type PCIE --show_links
Valid PCIe Links
----------------
Legend : depth ,pcie_index ,node ,port ,bdf ,link_status
Link 1 : 0 ,0 ,0 ,0 ,0001:00:00.0 ,UP
Example: PCIe counters
# mlxlink -d /dev/mst/mt12181_pciconf0 --port_type PCIE --show_counters
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 32G PAM-4 Gen6 (32G PAM-4 Gen6)
Link Width Active (Enabled) : 16X (16X)
Management PCIe Performance Counters Info
-----------------------------------------
RX Errors : 0
TX Errors : 1
CRC Error dllp : 0
CRC Error tlp : 0
FEC Correctable Error count : 508
FEC Uncorrectable Error count : 0
First BER (FBER) : 1E-14
Effective ber : 15E-255
Flit Error Ratio : 0
Example: eye opening
# mlxlink -d /dev/mst/mt12181_pciconf0 --port_type PCIE --show_eye
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 0, 0, 0
Link Speed Active (Enabled) : 32G PAM-4 Gen6 (32G PAM-4 Gen6)
Link Width Active (Enabled) : 16X (16X)
EYE Opening Info (PCIe)
-----------------------
FOM Mode : SLRG_FOM_MODE_EYE_OPEN
Lane : 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15
Initial FOM : 1248, 1155, 1232, 1320, 1239, 1288, 1348, 1338, 1310, 1224, 1281, 1187, 1204, 1357, 1404, 1363
Last FOM : 1248, 1155, 1232, 1320, 1239, 1288, 1348, 1338, 1310, 1224, 1281, 1187, 1204, 1357, 1404, 1363
Pass/Fail Criteria
SLRED
mlxlink -d [device] --port_type PCIE --margin
ConnectX-4/ConnextX-4 Lx/ConnectX-5/Bluefield
Gen3
|
Gen3 |
|
|
Eye width |
f(Height, Width) |
|
0 < Eye width< 50 |
FAIL |
|
50< Eye width< 60 |
pass external, fail internal |
|
60< Eye width |
PASS |
Gen4
|
Gen4 |
|
|
Eye Margin |
f(Height, Width) |
|
0 < Eye Grade < 1500 |
FAIL |
|
1500 < Eye Grade < 4500 |
Pending on BER criteria |
|
4500 < Eye Grade |
PASS |
ConnectX-6/ConnectX-6 Dx/ConnectX-6 Lx/BlueField-2
Gen3
|
Gen3 |
|
|
Eye Grade |
FOM(figure of merit) |
|
0 < Eye Grade < 700 |
FAIL |
|
700 < Eye Grade < 2300 |
Pending on BER criteria |
|
2300 < Eye Grade |
PASS |
Gen4
|
Gen4 |
|
|
Eye Margin |
FOM |
|
0 < Eye Grade < 69 |
FAIL |
|
70< Eye Grade < 99 |
Pending on BER criteria |
|
100< Eye Grade |
PASS |
Gen5
|
Gen4 |
|
|
Eye Margin |
FOM |
|
0 < Eye Grade < 49 |
FAIL |
|
50< Eye Grade < 69 |
Pending on BER criteria |
|
70< Eye Grade |
PASS |
ConnectX-7/BlueField-3
Gen4/Gen3
|
Gen4 |
|
|
Last FOM |
Eye Margin |
|
0 < Last FOM < 69 |
FAIL |
|
70< Last FOM < 99 |
Pending on BER criteria |
|
100< Last FOM |
PASS |
Gen5
|
Gen5 |
|
|
Last FOM(figure of merit) |
Eye Margin |
|
0 < Last FOM < 65 |
FAIL |
|
65< Last FOM < 95 |
Pending on BER criteria |
|
95< Last FOM |
PASS |
NVLink5 Switch ASIC/ConnectX-8/ConnectX-9
Gen3
|
Gen3 |
|
|
Last FOM(figure of merit) |
Eye Margin |
|
0 < Last FOM < 1499 |
FAIL |
|
1500< Last FOM < 1881 |
BER 1e-15 |
|
1881< Last FOM |
PASS |
Gen4
|
Gen4 |
|
|
Last FOM(figure of merit) |
Eye Margin |
|
0 < Last FOM < 1499 |
FAIL |
|
1500< Last FOM < 1881 |
BER 1e-15 |
|
1881< Last FOM |
PASS |
Gen5
|
Gen5 |
|
|
Last FOM(figure of merit) |
Eye Margin |
|
0 < Last FOM < 999 |
FAIL |
|
1000< Last FOM < 1400 |
BER 1e-15 |
|
1400< Last FOM |
PASS |
Gen6
|
Gen6 |
|
|
Last FOM(figure of merit) |
Eye Margin |
|
0 < Last FOM < 200 |
FAIL |
|
200< Last FOM < 400 |
FBER 1e-9 |
|
400< Last FOM |
PASS |
PCIE Error Injection
This test feature allows errors injection over the PCI links. It is used to verify that the system can handle the PCIe errors, which rarely occur in regular usage.
The ConnectX-7 device includes testability features that can be configured to act as an error injection ‘exerciser’ in order to test other components in the system. This is supported when the ConnectX-7 is used as a PCIe switch.
-
This is a PCIe related feature that should be run over PCIe links only (
--port_type PCIE) with specific depth, PCIe index and node (DPN) -
If the DPN is not provided, the tool will take the default values - 0,0 and 0, respectively
-
The mapping between the BDF and its DPN can be found by executing the
show_linkscommand (see example below)
Error Types
|
ID |
Error Type |
Description |
Unit |
Additional Parameters ( |
Advanced Error Reporting Flag Set by This Error |
|
0 |
ABORT |
Cancels the current pending error, if exists. |
NA |
NA |
NA |
|
1 |
BAD_DLLP_LCRC |
Flips a bit in the LCRC of the next “ |
Packets |
NA |
Bad DLLP Status |
|
2 |
BAD_TLP_LCRC |
Flips a bit in the LCRC of the next “ through the port. The packets are VDM TLPs that are sent by the port to the destination BDF - “ |
Packets |
NA |
Bad TLP Status |
|
3 |
BAD_TLP_ECRC |
Flips a bit in the ECRC of the next “error_duration” TLPs that are transmitted through the port. The packets are VDM TLPs that are sent by the port to the destination BDF - “dbdf”. |
Packets |
NA |
ECRC Error Status |
|
4 |
ERR_MSG |
Sends an error signaling message to the RC. |
Packets |
Parameter 0: message type 0 - Correctable 1 - Nonfatal 2 - Fatal |
ERR_COR Received / Non-Fatal Error Messages Received / Fatal Error Messages Received |
|
5 |
MALFORMED_ TLP |
Sends an “error_duration” PM_ACTIVE_STATE_NACK message to the destination BDF - “dbdf” with TC=1 instead of 0. |
Packets |
NA |
Malformed TLP Status |
|
6 |
POISONED_TLP |
Sends an “error_duration” VDMs with data to the destination BDF - “dbdf” with EP = 1. |
Packets |
NA |
Poisoned TLP Received |
|
7 |
UNEXPECTED_CPL |
Sends “error_duration” completions to the destination BDF - “dbdf” with 0xff tag. |
Packets |
NA |
Unexpected Completion Status |
|
8 |
ACS_VIOLATION |
Sends “error_duration” VDMs to the destination BDF - “dbdf” with source_bdf=0. |
Packets |
NA |
ACS Violation Status |
|
100 |
SURPRISE_LINK_DOWN |
Sets a port state to DETECT. |
NA |
NA |
Surprise Down Error Status |
|
101 |
RECEIVER_ERROR |
Sends a clock instead of data for “error_duration” usecs. A value of 0 in ‘error_duration’ means that this error must be toggled by the firmware as fast as possible. |
uSec |
NA |
Receiver Error Status |
PCIe Error Injection Inputs
The following values should be provided in the error injection command line. Some values may be optional according to the error type.
|
Input |
Command Line Flag |
Description |
Obligatory |
Default |
|
Error Type |
|
Error type according to the table above. |
Yes |
- |
|
Error Duration |
|
The minimal number of packets with this error that will be sent, or the minimal amount of time that this error state would be applied. |
No |
1 |
|
Injection Delay |
|
Delay in microseconds before the error is applied.
|
No |
0 |
|
Destination BDF |
|
Destination BDF. Relevant for some of the errors that require packet generation. See error table above. |
No |
0:00.0 |
|
Additional Parameters |
|
Additional parameters according to the error type.
|
No |
0 |
Mlxlink will trigger the firmware to start the error injection process by providing the --pcie_error_injection flag with the requested configuration parameters.
Note that the command returns immediately, but the error injection can take longer to complete (according to the error duration and injection delay inputs).
When the tool is run without the parameters above, it will query the error injection state – Whether it is ready to start a new error injection, or it is in the middle of the previous injection.
Usage Example
Start the process by performing error injection with error type UNEXPECTED_CPL.
This example shows how to start the error injection by sending 5 unexpected completion packets. The packets (of error type UNEXPECTED_CPL (id 7)) are directed from BDF 05:00.0 to BDF 06:0.0 after 500µs of sending the command in the following environment:
|
PCIe Component |
BDF |
|---|---|
|
Root port |
00:01.0 |
|
(exerciser) PCIe Switch Upstream port |
01:00.0 |
|
(exerciser) PCIe Switch Downstream port |
05:00.0 |
|
Endpoint |
06:00.0 |
To get the related depth, pcie_index and node for the specific BDF 05:00.0, the show_links command should be executed as follows:
Show Links
mlxlink -d /dev/mst/mt4129_pciconf0 --port_type PCIE --show_links
Valid PCIe Links
----------------
Legend : depth ,pcie_index ,node ,port ,bdf
Link 1 : 0 ,0 ,0 ,0 ,01:00.0
Link 2 : 3 ,0 ,0 ,60 ,05:00.0
In this case, the depth, pcie_index, and node flags for the downstream port should be 3, 0, and 0, respectively.
Then, the following command can be executed to start the process:
Start PCIe Error Injection
mlxlink -d /dev/mst/mt4129_pciconf0 --port_type PCIE --depth 3 --pcie_index 0 --node 0 --pcie_error_injection --error_type UNEXPECTED_CPL --error_duration 5 --dbdf 06:00.0 --injection_delay 500
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
PCIe error injection might cause a PCIe bus failures or a system hang
Do you want to continue? yes
Starting PCIe Error Injection...
Query Error Injection Status
After sending the configuration command, the progress of the process can be checked by executing the tool with the pcie_error_injection flag only:
Query PCIe Error Injection Status 1
mlxlink -d /dev/mst/mt4129_pciconf0 –-port_type PCIE --depth 3 --pcie_index 0 --node 0 –-pcie_error_injection
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
PCIe Error Injection Info
-------------------------
Error Injection Status : In progress
Error Injection Type : UNEXPECTED_CPL
Error Injection Duration : 5 Packets
Once the process is complete, the output will be changed to "ready". This means that another error injection request can be submitted:
Query PCIe Error Injection Status 2
mlxlink -d /dev/mst/mt4129_pciconf0 –-port_type PCIE --depth 3 --pcie_index 0 --node 0 –-pcie_error_injection
PCIe Operational (Enabled) Info
-------------------------------
Depth, pcie index, node : 3, 0, 0
Link Speed Active (Enabled) : 16G-Gen 4 (16G-Gen 4)
Link Width Active (Enabled) : 16X (16X)
PCIe Error Injection Info
-------------------------
Error Injection Status : Ready
Error Injection Type : N/A
Error Injection Duration : N/A
Last updated: