Networking InfiniBand Cluster Bring-up Procedure

Zero Touch Provisioning for NDR

Zero Touch Provisioning (ZTP) automates initial switch configuration at boot time, allowing automatic OS image upgrade, configuration file application, and docker container deployment — without manual CLI intervention.

ZTP is enabled by default and is based on DHCP. It runs automatically on all management interfaces when no configuration has been saved.

Prerequisites

  • A DHCP server (v4 or v6) reachable from the switch management interface

  • A provisioning server (SCP/SFTP/HTTP/TFTP) hosting the required files

  • Prepared files as needed:

    • OS image (e.g. image-X86_64-3.x.xxxx.img) — review the upgrade matrix in the MLNX-OS release notes before selecting an image

    • Configuration file — text file created from the output of show running-config

    • Docker container image (optional, e.g. ubuntu.img.gz)

Only a textual configuration file is supported. Only one docker container can be launched via ZTP.

Step 1 — Prepare the Configuration File

  1. On a reference switch (or from a template), export the running config:

    switch (config) # show running-config
    
    
  2. Save the output to a text file (e.g. switch.conf).

  3. Critical: Append these two lines at the end of the configuration file to save the config and disable ZTP after provisioning:

    no zero-touch suppress-write
    configuration write
    
    

    Without these lines, configuration write is suppressed during ZTP, meaning the config won’t be saved and ZTP will remain active after reboot.

Step 2 — Host the Files on a Provisioning Server

Upload the OS image, configuration file, and docker image (if used) to your provisioning server. Supported protocols: SCP, SFTP, HTTP, TFTP.

Step 3 — Configure the DHCP Server

DHCPv4

Set options 66 and 67. Option 66 contains the URL prefix (server path without filename); option 67 contains the filenames in a fixed comma-separated order: <image>, <config>, <docker>.

host switch-1 {
    hardware ethernet E4:1D:2D:5B:72:80;
    fixed-address 10.0.1.13;
    option tftp-server-name "scp://<user>:<password>@<server>/ztp/, scp://<user>:<password>@<server>/ztp/, scp://<user>:<password>@<server>/ztp/";
    option bootfile-name "image-X86_64-3.11.3002.img, switch.conf, ubuntu.img.gz";
}

DHCPv6

Set option 58. Each item is a complete URL (not prefix + filename), in the same comma-separated order:

host switch-1 {
    option dhcp6.bootfile-url "scp://<user>:<password>@[<server-ipv6>]/ztp/image-X86_64-3.11.3002.img, scp://<user>:<password>@[<server-ipv6>]/ztp/switch.conf, scp://<user>:<password>@[<server-ipv6>]/ztp/ubuntu.img.gz";
}

An item value can be empty (to skip that component), but the commas must not be omitted. For example, to apply only a config file: option bootfile-name ", switch.conf, ";

Step 4 — Boot the Switch

  1. Power on or reboot the switch (ZTP runs automatically when no config is saved).

  2. The switch sends DHCP requests on all management interfaces with identifying information (model, serial number, MAC, OS version) via vendor-specific options.

  3. The DHCP server responds with file URLs.

  4. ZTP executes in order:

    • Image: Downloads and installs the OS image if it differs from the active version. If a reboot is required, ZTP resumes from step 1 after reboot (no duplicate upgrade).

    • Configuration: Fetches and applies the configuration file.

    • Docker: Fetches, loads, and starts the docker container.

  5. If any step fails, the switch waits 1–20 seconds (random) and retries, up to 10 attempts.

  6. ZTP progress is printed to the console and active SSH sessions.

Step 5 — Verify ZTP Status

switch (config) # show zero-touch

Example output:

Zero-Touch status:
  Active: yes
  Status: Waiting for zero-touch start
  Suppress-write: no
  Configured by zero-touch: no
  Configuration changed after zero-touch: no

Disabling and Re-enabling ZTP

Action

Command

Disable ZTP

configuration write (any user-initiated config save disables ZTP)

Re-enable ZTP

reset factory (clears all config and reboots into ZTP)

Abort ZTP in progress

zero-touch abort

reset factory is the only way to re-enable ZTP after it has been disabled.

Modular Switch Systems (e.g. CS7500)

For modular systems, both management nodes run ZTP independently with status synchronization:

  • Both nodes must target the same software image version

  • Failure or abort on one node affects both

  • Use #<CHASSIS_MASTER> / #</CHASSIS_MASTER> tags in the config file to mark commands that should only execute on the master node:

    #<CHASSIS_MASTER>
       chassis ha bip 10.7.146.34 /24
    #</CHASSIS_MASTER>
    
    

Upgrading from Non-ZTP Versions

When upgrading from a pre-ZTP MLNX-OS version, ZTP is automatically disabled to avoid mixing existing configuration with new ZTP-applied configuration.

Review the upgrade matrix section in the MLNX_OS release notes for correct versions.


Last updated: