This feature is only supported on EDR switches.
As it is becoming increasingly complex to manage networks, and network administrators need more tools to understand network behavior, it is necessary to provide basic information about network performance, identify network bottlenecks, and provide information for the purposes of network optimization and future planning.
Therefore, network administrators are required to constantly review network port behavior, record port buffer consumption, and identify shortage in buffer resources and record flows which lead to the excessive buffer consumption. MLNX-OS provides the following mechanisms to perform these tasks:
-
Sampling (histograms)—a network administrator can enable a sampling of the port buffer occupancy, record occupancy changes over time, and provide information for different levels of buffer occupancy, and amount of time the buffer has been occupied during the observation period.
-
Thresholds—thresholds may be enabled per port to record the network time when port buffer occupancy crosses the defined threshold and when buffer occupancy drops below it.
-
Flow recording—a record of the most active flows which cause an excessive usage of the port buffers may be kept. Once enabled, the system may identify flow patterns and present a user with a list of flows, based on which a network administrator can rearrange distribution of the data flows in the network and minimize data loss.
Buffer Histograms and Thresholds Commands
protocol telemetry
|
|
protocol telemetry no protocol telemetry Unhides telemetry config CLIs. The no form of the command hides telemetry config CLIs. |
|
|
Syntax Description |
N/A |
|
|
Default |
Hidden |
|
|
Configuration Mode |
config |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config) # protocol telemetry |
|
|
Related Commands |
|
|
|
Notes |
|
|
telemetry shutdown
|
|
telemetry shutdown
Disables the telemetry protocol, threshold detection, and histogram fetching for all sampling enabled interfaces without changing any internal configuration.
|
|
|
Syntax Description |
N/A |
|
|
Default |
Disabled |
|
|
Configuration Mode |
config |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config) # no telemetry shutdown |
|
|
Related Commands |
protocol telemetry |
|
|
Notes |
|
|
telemetry sampling
|
|
interface ib <slot>/<port> telemetry sampling
Enables sampling (histogram fetching) for a specific InfiniBand interface.
|
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
config interface ib |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config interface ib 1/1) # telemetry sampling |
|
|
Related Commands |
protocol telemetry |
|
|
Notes |
|
|
telemetry sampling log
|
|
telemetry sampling log <time>
Enables the log interval value (histogram fetching) from device.
|
|
|
Syntax Description |
time |
Input range: 100-60000 (in msec) |
|
Default |
1000 millisecond |
|
|
Configuration Mode |
config |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config) # telemetry sampling log 1000 |
|
|
Related Commands |
protocol telemetry |
|
|
Notes |
|
|
telemetry threshold
|
|
telemetry threshold
Enables telemetry threshold on hardware.
|
|
|
Syntax Description |
N/A |
|
|
Default |
false |
|
|
Configuration Mode |
config interface ib |
|
|
History |
3.6.4006 |
|
|
Example |
switch (config interface ib 1/1) # telemetry threshold |
|
|
Related Commands |
|
|
|
Notes |
|
|
telemetry threshold level
|
|
telemetry threshold level <level>
Configures threshold level in hardware per port.
|
|
|
Syntax Description |
level |
Input range: 96-1000000 (in bytes and in increments of 96) |
|
Default |
69984 |
|
|
Configuration Mode |
config interface ib |
|
|
History |
3.6.4006 |
|
|
Example |
switch (config interface ib 1/1) # telemetry threshold level 288 |
|
|
Related Commands |
|
|
|
Notes |
|
|
telemetry threshold log
|
|
telemetry threshold log
Enables logging of threshold events in syslog.
|
|
|
Syntax Description |
N/A |
|
|
Default |
Disabled |
|
|
Configuration Mode |
config |
|
|
History |
3.6.4006 |
|
|
Example |
switch (config) # telemetry threshold log |
|
|
Related Commands |
|
|
|
Notes |
|
|
telemetry threshold record
|
|
telemetry threshold record
Enables top talker configuration.
|
|
|
Syntax Description |
N/A |
|
|
Default |
Disabled |
|
|
Configuration Mode |
config interface ib |
|
|
History |
3.6.6105 |
|
|
Example |
switch (config interfaces ib 1/2) # telemetry threshold record |
|
|
Related Commands |
clear telemetry threshold record
|
|
|
Notes |
|
|
telemetry threshold syslog
|
|
telemetry threshold syslog <time>
The command sets threshold events logging rate on per hour basis.
|
|
|
Syntax Description |
time |
Max rate per hour
|
|
Default |
100 |
|
|
Configuration Mode |
config |
|
|
History |
3.6.4006 |
|
|
Example |
switch (config) # telemetry threshold syslog 400 |
|
|
Related Commands |
|
|
|
Notes |
|
|
clear telemetry
|
|
clear telemetry {threshold | sampling} [interface ib <port-id>] Clears telemetry data. |
|
|
Syntax Description |
port-id |
InfiniBand interface ID |
|
Default |
N/A |
|
|
Configuration Mode |
config interface ib |
|
|
History |
3.6.5000 |
|
|
Example |
switch (config interface ib 1/12) # clear telemetry threshold level 288 |
|
|
Related Commands |
|
|
|
Notes |
|
|
clear telemetry threshold
|
|
clear telemetry threshold [interface <type> <if>] Clears threshold and top talker data. |
|
|
Syntax Description |
type |
Available values:ib |
|
Default |
N/A |
|
|
Configuration Mode |
config |
|
|
History |
3.6.6105 |
|
|
Example |
switch (config) # clear telemetry threshold interface ib 1/34-1/36 |
|
|
Related Commands |
|
|
|
Notes |
|
|
clear telemetry threshold record
|
|
clear telemetry threshold record [interface ib <if>] Clears top talker data. |
|
|
Syntax Description |
if |
InfiniBand interface ID |
|
Default |
N/A |
|
|
Configuration Mode |
config |
|
|
History |
3.6.6105 |
|
|
Example |
switch (config) # clear telemetry threshold record interface ib 1/34-1/36 |
|
|
Related Commands |
telemetry threshold record
|
|
|
Notes |
|
|
stats export csv telemetry
|
|
stats export csv telemetry <slot>/<port>[/<subport>][filename <name>] [after * *] [before * *] Exports histograms collected by stats to a csv file. |
|
|
Syntax Description |
slot/port |
Port number |
|
subport |
Subport number to be used if a port is split |
|
|
Default |
N/A |
|
|
Configuration Mode |
Any command mode |
|
|
History |
3.6.3004 |
|
|
3.9.0500 |
Updated example |
|
|
Example |
switch (config) # stats export csv telemetry 1/1/4-ucast after 2020/03/16 10:54:58 before 2020/03/16 11:16:24Generated report file: telemetry-20200316-111704.csv |
|
|
Related Commands |
|
|
|
Notes |
|
|
file stats telemetry delete
|
|
file stats telemetry delete <filename> Deletes the given .csv file created by “stats export” command to user directory. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
config |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config) # file stats telemetry delete telemetry-20171006-102158.csv |
|
|
Related Commands |
|
|
|
Notes |
|
|
file stats telemetry delete latest
|
|
file stats telemetry delete latest Delete the latest stats telemetry file. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
Configure terminal |
|
|
History |
3.8.1000 |
|
|
Example |
(config) # file stats telemetry delete latest |
|
|
Related Commands |
file stats telemetry delete <file_name>
|
|
|
Notes |
|
|
file stats telemetry delete all
|
|
file stats telemetry delete all Deletes all stats telemetry files from machine. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
Configure terminal |
|
|
History |
3.8.1000 |
|
|
Example |
(config) # file stats telemetry delete all |
|
|
Related Commands |
file stats telemetry delete <file_name>
|
|
|
Notes |
|
|
file stats telemetry upload
|
|
file stats telemetry upload Uploads .csv file created by “stats export” command to user directory. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
config |
|
|
History |
3.6.3004 |
|
|
Example |
switch (config) # file stats telemetry upload telemetry-20170119-102715.csv scp://username:password@server//directoryPassword (if required): ****** |
|
|
Related Commands |
|
|
|
Notes |
|
|
file stats telemetry upload latest
|
|
file stats telemetry upload latest Upload the latest stats telemetry file to a remote host. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
Configure terminal |
|
|
History |
3.8.1000 |
|
|
Example |
(config) # file stats telemetry upload latest scp://user:pass@10.135.155.8/tmp |
|
|
Related Commands |
file stats telemetry upload <file_name>
|
|
|
Notes |
|
|
file stats telemetry upload all
|
|
file stats telemetry upload all
Upload all stats telemetry files to a remote host. |
|
|
Syntax Description |
||
|
Default |
N/A |
|
|
Configuration Mode |
Configure terminal |
|
|
History |
3.8.1000 |
|
|
Example |
(config) # file stats telemetry upload all scp://user:pass@10.135.155.8/tmp |
|
|
Related Commands |
file stats telemetry upload <file_name>
|
|
|
Notes |
|
|
show telemetry
|
|
show telemetry Displays the global configuration of telemetry properties. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
config |
|
|
History |
3.6.4000 |
|
|
Example |
||
|
switch (config) # show telemetry Telemetry Status : Enabled H/W Sampling Interval(nsec) : 512 S/W Sampling Interval(ms) : 1000 Threshold Logging : Disabled Threshold Logging(rate per hour) : 100 -------------------------------------------------------------------------------------------- Interface Sampling Threshold Record Level (bytes) -------------------------------------------------------------------------------------------- IB1/1 Disabled Enabled Enabled 100 (96) IB1/2 Disabled Enabled Enabled 100 (96) IB1/3 Disabled Disabled Disabled N/A IB1/4 Disabled Disabled Disabled N/A IB1/5 Disabled Disabled Disabled N/A IB1/6 Disabled Disabled Disabled N/A IB1/7 Disabled Disabled Disabled N/A ... IB1/36 Disabled Disabled Disabled N/A |
||
|
Related Commands |
|
|
|
Notes |
|
|
show telemetry threshold record
|
|
show telemetry threshold record [interface ib <interface-id> | <interface-id-range>] Displays top talker events for all configured ports. |
|
|
Syntax Description |
N/A |
|
|
Default |
N/A |
|
|
Configuration Mode |
Any command mode |
|
|
History
|
3.6.4006 |
|
|
3.6.6105 |
Updated example |
|
|
3.6.8100 |
Updated example |
|
|
|
||
|
Example |
||
|
switch (config) # show telemetry threshold record interface ib 1/11-1/12 ----------------------------------------------------------------------------------------------------------------- Event-id Date Time Port Level Duration(100 usec) Repeated DestQP DLID SLID Percent(%) ----------------------------------------------------------------------------------------------------------------- 1 07/10/18 14:00:31 IB 1/11 69984 48749.77 1 2741 29 32 62.30 2 07/10/18 14:01:47 IB 1/11 69984 63936.16 1 2745 29 32 54.55 |
||
|
Related Commands |
clear telemetry threshold |
|
|
Notes |
|
|
show telemetry sampling interface ib
|
|
show telemetry sampling interface ib <slot>/<port> Displays telemetry histogram samples for a specific InfiniBand interface. |
|
|
Syntax Description |
slot/port |
InfiniBand interface number |
|
Default |
N/A |
|
|
Configuration Mode |
Any command mode |
|
|
History |
3.6.3004 |
|
|
Example |
||
|
switch (config) # show telemetry sampling interface ib 1/32 ------------------------------------------------------------------------------------------------------------------------------------- Telemetry histogram: IB1/32 System-time Bin sizes (128 nsec tx buffer was occupied in bytes range) ------------------------------------------------------------------------------------------------------------------------------------- 02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120< 12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0 12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0 12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0 12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0 12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0 12:19:08.42761 160 9302 7803040 0 0 0 0 0 0 0 12:19:09.42915 304 9369 7802829 0 0 0 0 0 0 0 12:19:10.43071 96 8906 7803500 0 0 0 0 0 0 0 12:19:11.43215 463 8907 7803132 0 0 0 0 0 0 0 12:19:12.43369 256 8571 7803675 0 0 0 0 0 0 0 |
||
|
Related Commands |
|
|
|
Notes |
If the requested entries are more than what the DB contains, it prints the amount in the table. |
|
show telemetry sampling interface ib last
|
|
show telemetry sampling interface ib <slot>/<port> last <num_of_entries> Displays fetched unicast histogram details for an InfiniBand interface. |
|
|
Syntax Description
|
slot/port |
InfiniBand interface number |
|
num_of_entries |
Range: 0-1000 |
|
|
Default |
N/A |
|
|
Configuration Mode |
Any command mode |
|
|
History |
3.6.3004 |
|
|
Example |
||
|
switch (config) # show telemetry sampling interface ib 1/36 last 20 Legend: 2976 bytes - between 0 - 2976 of tx bytes buffer consumed 35744 bytes - between 2977 - 35744 of tx bytes buffer consumed ------------------------------------------------------------------------------------------------------------------------------------- Telemetry histogram: IB1/36 System-time Bin sizes (128 nsec tx buffer was occupied in bytes range) ------------------------------------------------------------------------------------------------------------------------------------- 02/09/17 <2976 35744 68512 101280 134048 166816 199584 232352 265120 265120< 12:19:03.41948 1883 8538 7802080 0 0 0 0 0 0 0 12:19:04.42107 830 9001 7802670 0 0 0 0 0 0 0 12:19:05.42249 96 9705 7802700 0 0 0 0 0 0 0 12:19:06.42388 32 9035 7803434 0 0 0 0 0 0 0 12:19:07.42573 80 9461 7802960 0 0 0 0 0 0 0 |
||
|
Related Commands |
|
|
|
Notes |
If requested entries are more than what the DB contains, it prints the amount in the table. |
|
show files stats telemetry
|
|
show files stats telemetry [filename] Displays all files created by the command “stats export csv telemetry”. |
|
|
Syntax Description |
filename |
Displays stats for the specified file |
|
Default |
N/A |
|
|
Configuration Mode |
Any command mode |
|
|
History
|
3.6.3004 |
|
|
3.6.8008 |
Updated example |
|
|
Example |
switch (config) # show files stats telemetry telemetry-20180527-102715.csvHostname :test-switchReport :telemetry histogramTime lower bound(UTC) :2018/05/28 05:58:10Time upper bound(UTC) :2018/05/28 05:58:25Export time(UTC) :2018/05/28 06:00:06Time lower bound :2018/05/28 08:58:10 +0300Time upper bound :2018/05/28 08:58:25 +0300Export time :2018/05/28 09:00:06 +0300System version :X86_64 sys_test 2018-05-15 04:02:13 x86_64 |
|
|
Related Commands |
stats export csv telemetry |
|
|
Notes |
|
|
Last updated: