QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual
QM97XX 1U NDR 400Gbps InfiniBand Switch Systems User Manual
Breadcrumbs

System Bring-Up of Managed Systems


The bring-up procedures described in this section do not apply to unmanaged/externally managed systems. Such systems are ready for operation after power-on.

In order to query the system, perform firmware upgrade or other firmware operation. Refer to the latest Mellanox Firmware tools (MFT) located on https://network.nvidia.com/products/adapter-software/firmware-tools/.

In order to obtain the firmware version of the externally managed system:

  1. Run the following command from a host:

    # flint -d <device> q


  2. Compare the results of this command with the latest version for your system posted on https://network.nvidia.com/products/adapter-software/firmware-tools/.

  3. If the current version is not the latest version, follow the directions in the MFT User Manual to burn the new firmware.

Configuring Network Attributes

The procedures described in this chapter assume that you have already installed and powered-on the system according to the instructions in this document. Since the system comes with a pre-configured DHCP, you may find the explanation in Disable Dynamic Host Configuration Protocol (DHCP) sufficient. In case manual configuration is required, please refer to the instructions in Manual Host Configuration.

Disable Dynamic Host Configuration Protocol (DHCP)

DHCP is used for automatic retrieval of management IP addresses.

If a user connects through SSH, runs the wizard and turns off DHCP, the connection is immediately terminated, as the management interface loses its IP address. In such a case, the serial connection should be used.

<localhost># ssh admin@<ip-address>

Mellanox MLNX-OS Switch Management
Password:
Mellanox Switch
Mellanox configuration wizard
Do you want to use the wizard for initial configuration? yes
Step 1: Hostname? [my-switch]
Step 2: Use DHCP on mgmt0 interface? [yes] no
<localhost>#

Manual Host Configuration

To perform initial configuration of the system:

  1. Connect a host PC to the Console RJ45 port of the system, using the supplied harness cable (DB9 to RJ45). 

    Make sure to connect to the Console RJ45 port, and not to the (Ethernet) MGT port. Pay attention to the icons:

    Console RJ45

    image2018-11-14_10-2-54.png

    Ethernet MGT

    image2018-11-13_16-13-1.png



  2. Configure a serial terminal program (for example, HyperTerminal, minicom, or Tera Term) on your host PC with the settings described in the table below. Once you perform that, you should get the CLI prompt of the system.

    Serial Terminal Program Configuration

    Parameter

    Setting

    Baud Rate

    115200

    Data bits

    8

    Stop bits

    1

    Parity

    None

    Flow Control

    None


  3. The boot menu is prompted.  

    ...                                                                             .
This terminal is not active for input or output while booting.


    Boot Menu                                                                   .

-------------------------------------------------------------------
0: <image #1>
1: <image #2>
-------------------------------------------------------------------

      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected image or 'p' to enter a
      password to unlock the next set of features.

    Highlighted entry is 0:


    Select “0” to boot with software version installed on partition #1.
    Select “1” to boot with software version installed on partition #2.

    The boot menu features a countdown timer. It is recommended to allow the timer to run out by not selecting any of the options.

  4. Login as admin and use admin as password. If the machine is still initializing, you might not be able to access the CLI until initialization completes. As an indication that initialization is ongoing, a countdown of the number of remaining modules to be configured is displayed in the following format: “<no. of modules> Modules are being configured”.

  5. Go through the Switch Management configuration wizard.

    IP Configuration by DHCP

    Wizard Session Display (Example)

    Comments

    Do you want to use the wizard for initial configuration? yes

    You must perform this configuration the first time you operate the switch or after resetting the switch to the factory defaults. Type “y” and then press <Enter>.

    Step 1: Hostname? [switch-1]

    If you wish to accept the default hostname, then press <Enter>. Otherwise, type a different hostname and press <Enter>.

    Step 2: Use DHCP on mgmt0 interface? [yes]

    Perform this step to obtain an IP address for the switch. (mgmt0 is the management port of the switch.)
    - If you wish the DHCP server to assign the IP address, type “yes” and press <Enter>.

    If you type “no” (no DHCP), then you will be asked whether you wish to use the “zeroconf” configuration or not. If you enter “yes” (yes Zeroconf), the session will continue as shown in the "IP zeroconf configuration" table.

    If you enter “no” (no Zeroconf), then you need to enter a static IP, and the session will continue as shown in the "Static IP configuration" table.

    Step 3: Enable IPv6 [yes]

    Perform this step to enable IPv6 on management ports.

    If you wish to enable IPv6, type “yes” and press <Enter>.

    If you enter “no” (no IPv6), then you will automatically be referred to Step 5.

    Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface

    Perform this step to enable StateLess address autoconfig on external management port.

    If you wish to enable it, type “yes” and press <Enter>.

    If you wish to disable it, enter “no”.

    Step 5: Use DHCPv6 on mgmt0 interface? [yes]

    Perform this step to enable DHCPv6 on the MGMT0 interface.

    Step 6: Update time?

    Perform this step to change the time configured. Press enter to leave the current time.

    Step 7: Enable password hardening?

    Perform this step to enable/disable password hardening on your machine. If enabled, new passwords will be checked upon configured restrictions.

    If you wish to enable it, type “yes” and press <Enter>.

    If you wish to disable it, enter “no”.

    Step 8: Admin password (Must be typed)? <new_password>

    To avoid illegal access to the machine, please type a password and then press <Enter>.

    Starting from the 3.8.2000 release, the user must type in the admin password upon initial configuration. Due to Senate Bill No. 327, this stage is required and cannot be skipped.

    Step 9: Confirm admin password? <new_password>

    Confirm the password by re-entering it. Note that password characters are not printed.

    Step 10: Monitor password (Must be typed)? <new_password>

    To avoid illegal access to the machine, please type a password and then press <Enter>.

    Starting from the 3.8.2000 release, the user must type in the admin password upon initial configuration. Due to Senate Bill No. 327, this stage is required and cannot be skipped.

    Step 11: Confirm monitor password? <new_password>

    Confirm the password by re-entering it. Note that password characters are not printed.

    You have entered the following information:Hostname: <switch name>Use DHCP on mgmt0 interface: yesEnable IPv6: yesEnable IPv6 autoconfig (SLAAC) on mgmt0 interface: yesEnable DHCPv6 on mgmt0 interface: noUpdate time: <current time>Enable password hardening: yesAdmin password (Enter to leave unchanged): (CHANGED)To change an answer, enter the step number to return to.Otherwise hit <enter> to save changes and exit.Choice: <Enter>Configuration changes saved.To return to the wizard from the CLI, enter the “configuration jump-start” command from configuration mode. Launching CLI...<switch name> [standalone: master] >

    The wizard displays a summary of your choices and then asks you to confirm the choices or to re-edit them.

    Either press <Enter> to save changes and exit, or enter the configuration step number that you wish to return to.

    To run the command “configuration jump-start” you must be in Config mode.

    IP Configuration by DHCP for Modular Switch Systems

    Wizard Session Display (Example)

    Comments

    Do you want to use the wizard for initial configuration? yes

    You must perform this configuration the first time you operate the switch or after resetting the switch to the factory defaults. Type “y” and then press <Enter>.

    Step 1: Hostname? [switch-1]

    If you wish to accept the default hostname, then press <Enter>. Otherwise, type a different hostname and press <Enter>.

    Step 2: Use DHCP on mgmt0 interface? [yes]

    Perform this step to obtain an IP address for the switch. (mgmt0 is the management port of the switch.)
    If you wish the DHCP server to assign the IP address, type “yes” and press <Enter>.

    If you type “no” (no DHCP), then you will be asked whether you wish to use the “zeroconf” configuration or not. If you enter “yes” (yes Zeroconf), the session will continue as shown in the IP zeroconf configuration" table.

    If you enter “no” (no Zeroconf), then you need to enter a static IP, and the session will continue as shown in the "Static IP configuration" table.

    Step 3: Enable IPv6 [yes]

    Perform this step to enable IPv6 on management ports.

    If you wish to enable IPv6, type “yes” and press <Enter>.

    If you enter “no” (no IPv6), then you will automatically be referred to Step 5.

    Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface

    Perform this step to enable StateLess address autoconfig on external management port.

    If you wish to enable it, type “yes” and press <Enter>.

    If you wish to disable it, enter “no”.

    Step 5: Use DHCPv6 on mgmt0 interface? [yes]

    Perform this step to enable DHCPv6 on the MGMT0 interface.

    Step 6: Admin password (Press <Enter> to leave unchanged)? <new_password>

    To avoid illegal access to the machine, please type a password and then press <Enter>.

    Step 7: Confirm admin password? <new_password>

    (this step only happens if you change the password)

    Confirm the password by re-entering it.

    Note that password characters are not printed.

    Step 9: HA Chassis Management IP netmask? (Example: [255.255.255.0])

    Perform this step to configure the box IPv4 netmask.

    If you wish to accept the default value, type “yes” and press <Enter>.

    Otherwise, enter the desired box IPv4 netmask

    Step 10: HA Chassis IPv6 address? (Example: [fdfd:fdfd:7:145::1000:4814])

    Perform this step to configure the box IPv6.

    If you wish to accept the default value, type “yes” and press <Enter>.

    Otherwise, enter the desired box IPv6

    Step 11: HA Chassis Management IPv6 masklen? (Example: [33])

    Perform this step to configure the box IPv6 masklen.

    If you wish to accept the default value, type “yes” and press <Enter>.

    Otherwise, enter the desired box IPv6 masklen.

    You have entered the following information:Hostname: <switch name>Use DHCP on mgmt0 interface: yesEnable IPv6: yesEnable IPv6 autoconfig (SLAAC) on mgmt0 interface: yesEnable DHCPv6 on mgmt0 interface: yesAdmin password (Enter to leave unchanged): (CHANGED)HA Chassis IP address: 10.6.166.200HA Chassis Management IP netmask: 255.255.255.0HA Chassis IPv6 address: fdfd:fdfd:7:145::1000:4814HA Chassis Management IPv6 masklen: 33To change an answer, enter the step number to return to.Otherwise hit <enter> to save changes and exit.Choice: <Enter>Configuration changes saved.To return to the wizard from the CLI, enter the “configuration jump-start” command from configuration mode. Launching CLI...<switch name> [standalone: master] >

    The wizard displays a summary of your choices and then asks you to confirm the choices or to re-edit them.

    Either press <Enter> to save changes and exit, or enter the configuration step number that you wish to return to.

    To run the command “configuration jump-start” you must be in Config mode.

    Static IP Configuration

    Wizard Session Display (Example)

    Do you want to use the wizard for initial configuration? y

    Step 1: Hostname? [switch-112126]Step 2: Use DHCP on mgmt0 interface? [yes] nStep 3: Use zeroconf on mgmt0 interface? [no]Step 4: Primary IP address? 192.168.10.4Mask length may not be zero if address is not zero (interface mgmt0)

    Step 5: Netmask? [0.0.0.0] 255.255.255.0Step 6: Default gateway? 192.168.10.1Step 7: Primary DNS server?Step 8: Domain name?Step 9: Enable IPv6? [yes] yesStep 10: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] noStep 11: Update time? [yyyy/mm/dd hh:mm:ss] Step 12: Enable password hardening? [yes] yesStep 13: Admin password (Enter to leave unchanged)?

    You have entered the following information:

    Hostname: switch-112126Use DHCP on mgmt0 interface: noUse zeroconf on mgmt0 interface: noPrimary IP address: 192.168.10.4Netmask: 255.255.255.0Default gateway: 192.168.10.1Primary DNS server:Domain name:Enable IPv6: yesEnable IPv6 autoconfig (SLAAC) on mgmt0 interface: noUpdate time: yyyy/mm/dd hh:mm:ssEnable password hardening: yesAdmin password (Enter to leave unchanged): (unchanged)

    To change an answer, enter the step number to return to.Otherwise hit <enter> to save changes and exit.

    Choice:Configuration changes saved.To return to the wizard from the CLI, enter the “configuration jump-start” command from configure mode. Launching CLI...<hostname>[standalone: master] >


    IP Zeroconf Configuration

    Wizard Session Display (Example)

    Configuration wizard Do you want to use the wizard for initial configuration? y Step 1: Hostname? [switch-112126]Step 2: Use DHCP on mgmt0 interface? [no]Step 3: Use zeroconf on mgmt0 interface? [no] yesStep 4: Default gateway? [192.168.10.1]Step 5: Primary DNS server?Step 6: Domain name?Step 7: Enable IPv6? [yes] yesStep 8: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no] noStep 9: Update time? [yyyy/mm/dd hh:mm:ss] Step 10: Admin password (Enter to leave unchanged)? You have entered the following information: Hostname: switch-112126Use DHCP on mgmt0 interface: noUse zeroconf on mgmt0 interface: yesDefault gateway: 192.168.10.1Primary DNS server:Domain name:Enable IPv6: yesEnable IPv6 autoconfig (SLAAC) on mgmt0 interface: yesUpdate time: yyyy/mm/dd hh:mm:ssEnable password hardening: yesAdmin password (Enter to leave unchanged): (unchanged) To change an answer, enter the step number to return to.Otherwise hit <enter> to save changes and exit. Choice: Configuration changes saved. To return to the wizard from the CLI, enter the “configuration jump-start”command from configure mode. Launching CLI...<hostname> [standalone: master] >


    IP Zeroconf Configuration for Modular Switch Systems

    Wizard Session Display (Example)

    Configuration wizard Do you want to use the wizard for initial configuration? y

    Step 1: Hostname? [switch-mgmt1]

    Step 2: Use DHCP on mgmt0 interface? [yes]

    Step 3: Enable IPv6? [yes]

    Step 4: Enable IPv6 autoconfig (SLAAC) on mgmt0 interface? [no]

    Step 5: Enable DHCPv6 on mgmt0 interface? [yes]

    Step 6: Admin password (Enter to leave unchanged)?

    Step 7: HA Chassis IP address: [10.6.166.200]

    Step 8: HA Chassis Management IP netmask: [255.255.255.0]

    Step 9: HA Chassis IPv6 address: [fdfd:fdfd:7:145::1000:4814]

    Step 10: HA Chassis Management IPv6 masklen: [33]


    You have entered the following information:

    1. Hostname: sw-mantaray-201-mgmt1

    2. Use DHCP on mgmt0 interface: yes

    3. Enable IPv6: yes

    4. Enable IPv6 autoconfig (SLAAC) on mgmt0 interface: no

    5. Enable DHCPv6 on mgmt0 interface: yes

    6. Admin password (Enter to leave unchanged): (unchanged)

    7. HA Chassis IP address: 10.6.166.200

    8. HA Chassis Management IP netmask: 255.255.255.0

    9. HA Chassis IPv6 address: fdfd:fdfd:7:145::1000:4814

    10. HA Chassis Management IPv6 masklen: 33


    To change an answer, enter the step number to return to.

    Otherwise hit <enter> to save changes and exit.


    Choice:

    Configuration changes saved.

    To return to the wizard from the CLI, enter the “configuration jump-start”command from configure mode. Launching CLI...<hostname> [standalone: master] >


  6. Check the mgmt0 interface configuration before attempting a remote (for example, SSH) connection to the switch. Specifically, verify the existence of an IP address. 

    switch # show interfaces mgmt0

Interface mgmt0 status:
  Comment         :
  Admin up        : yes
  Link up         : yes
  DHCP running    : yes
  IP address      : 10.12.67.34
  Netmask         : 255.255.0.0
  IPv6 enabled    : yes
  Autoconf enabled: no
  Autoconf route  : yes
  Autoconf privacy: no
  DHCPv6 running  : no
  IPv6 addresses  : 1

  IPv6 address:
    fe80::268a:7ff:fe53:3d8e/64

  Speed           : 1000Mb/s (auto)
  Duplex          : full (auto)
  Interface type  : ethernet
  Interface source: physical
  MTU             : 1500
  HW address      : 00:02:c9:11:a1:b2

  Rx:
   11700449  bytes
      55753  packets
          0  mcast packets
          0  discards
          0  errors
          0  overruns
          0  frame

  Tx:
   5139846  bytes
     28452  packets
         0  discards
         0  errors
         0  overruns
         0  carrier
         0  collisions
      1000  queue len


Configuring the Switch with ZTP

Zero-touch Provisioning (ZTP) automates initial configuration of switch systems at boot time. It helps minimize manual operation and reduce customer initial deployment cost.

For more information, please refer to the “Zero-touch Provisioning” section in the "Getting Started" chapter of the latest MLNX-OS User Manual.

Rerunning the Wizard

To rerun the wizard:

  1. Enter Config mode. Run:

    switch > enable
switch # config terminal


  2. Rerun the wizard. Run:

    switch (config) # configuration jump-start


Starting the Command Line (CLI) 

  1. Set up an Ethernet connection between the switch and a local network machine using a standard RJ-45 connector.

  2. Start a remote secured shell (SSH) to the switch using the command “ssh -l <username> <switch ip address>”. 

    rem_mach1 >  ssh -l <username> <ip address>


  3. Log into the switch (default username is admin, password admin).

  4. Read and accept the EULA when prompted.

  5. Once the following prompt appears, the system is ready to use. 

    NVIDIA MLNX-OS Switch Management

Password:
Last login: <time> from <ip-address>


NVIDIA Switch
Please read and accept the End User License Agreement located at:
https://www.mellanox.com/related-docs/prod_management_software/MLNX-OS_EULA.pdf
switch >


Remote Connection

Once the network attributes are set, you can access the CLI via SSH or the WebUI via HTTP/ HTTPs.

To access the CLI, perform the following steps:

  1. Set up an Ethernet connection between the system and a local network machine using a standard RJ45 cable.

  2. Start a remote secured shell (SSH) using the command: ssh -l <username> <IP_address>

    # ssh -l <username> <ip_address>
Mellanox MLNX-OS Switch Management 

Password:


  3. Login as admin (default username is admin, password is admin).

  4. Once you get the CLI prompt, you are ready to use the system.

For additional information about MLNX-OS, refer to the MLNX-OS User Manual located under https://docs.nvidia.com/networking/software/switch-software/index.html#mlnx-os-infiniband.

Last updated: