Networking InfiniBand Cluster Bring-up Procedure

Cable Validation

Cable validation is the process of validating the physical cable deployment against the expected topology defined during planning.

This process can be performed in parallel with deployment and/or between deployment phases to verify that the currently installed equipment matches the expected design. The tool is not dependent on a running Subnet Manager (SM) or on any functional InfiniBand (IB) communication; instead, it uses the out-of-band (OOB) management interfaces of the connected network devices.

This enables you to bring up the cluster gradually/incrementally and validate the deployment in smaller pieces rather than all at once.

Success Criterion: Every connection that should already be installed is reported as PASS . Remaining failures must correspond only to equipment or media that has not yet been installed.

Follow these steps to perform cable validation:

Steps 3–6 are the Setup & Validate wizard: Load Topology, Credentials, Pre-Check and Deploy Agents, then Validate.

  1. Deploy the CVT Collector

  2. Configure CVT

  3. Load the Planned Topology

  4. Set Credentials

  5. Run Pre-check and Deploy Agents

  6. Start Validation

  7. Review Results and Remediate

Step 1 - Deploy the CVT Collector

Install CVT on a host that can reach the management addresses of the switches to be validated over IPv4 or IPv6.

  • UFM Enterprise plugin: Install and start CVT from UFM Plugin Management, refresh the UFM GUI, and open Cable Validation from the left-hand menu.

  • Standalone: Run the collector container on a management-network host using host networking.

Deployment details are available in https://networking-docs.nvidia.com/cablevalidationtool/2.0.1/deploying-the-module and https://networking-docs.nvidia.com/cablevalidationtool/2.0.1/cvt-as-a-service.

  • Standalone UI: https://<collector-host>/cables_validation

  • UFM plugin: UFM GUI → Cable Validation

Sign in and start the bring-up service if the UI reports that it is not running. Select the fabric type: InfiniBand, XDR, or Ethernet.

From the collector host, verify reachability to every switch management address planned for validation. Unreachable management addresses will cause pre-check and agent deployment failures.

Step 2 - Configure CVT

Configure CVT before loading the topology or deploying agents. Agents download their image from the collector and send reports back to it, so the advertised collector address must be reachable from the switches.

Settings are stored in cvt_env.conf:

  • Standalone: <cable_bringup_root>/config/cvt_env.conf

  • UFM plugin: /config/cvt_env.conf

After editing the file, apply changes with supervisorctl restart cvt-service. A container environment variable overrides the same setting when the collector reads it. Agent settings in the [agent] section must be configured in the file so the collector and agents use the same intervals.

Management Reachability

If DEFAULT_AGENTS_INTERFACE_NAME is empty, CVT resolves the collector hostname. This can produce 127.0.0.1 or an interface that switches cannot reach. Check the startup log for Collector IP address that will be used by the agents, then verify that address from a switch.

Setting

Use when

DEFAULT_AGENTS_INTERFACE_NAME

The hostname-derived address is not reachable from switches. Set it to a reachable collector interface, such as eno3.

HOST_SPECIFIC_INTERFACE_NAME

Host agents must use a different collector interface than switch agents.

AGENTS_COLLECTOR_NAT_IP and AGENTS_COLLECTOR_NAT_PORT

A NAT or port-forward separates agents from the collector. If the port is empty, CVT uses APACHE_HTTPS_PORT, or port 443.

Pre-check connectivity timing is controlled by PING_COUNT, PING_TIMEOUT_SEC, and SSH_CONN_TIMEOUT.

Startup Behavior

For a manual bring-up through Setup & Validate, retain the defaults.

Setting

Default

Bring-up Behavior

STARTUP_TOPOLOGY

none

Load the topology in the UI. Alternatively use last or provide a supported topology file path.

STARTUP_CREDENTIALS_FILE

none

Leave credentials to the UI, or provide a supported .json, .ini, or .cfg file.

AUTO_START_VALIDATION

false

Set to true only when validation should start automatically after topology load.

Warning: Use the exact key STARTUP_CREDENTIALS_FILE in cvt_env.conf. Do not use the misspelled key STARTUP_CREDENTAILS_FILE found in the official CVT 2.0.1 documentation; the collector binary reads only STARTUP_CREDENTIALS_FILE and will ignore the misspelled key.

Agents check link state approximately every 10 seconds using CHECK_PORTS_INTERVAL. When there is no change, they send a periodic report approximately every 10 minutes using AGENT_COLLECT_INTERVAL, whose default is 600 seconds.

Step 3 - Load the Planned Topology

In Setup & Validate → Load Topology, load the planning file for the cluster.

For a new InfiniBand deployment, use a P2P workbook in Unified format. Enter each device’s management address in the IP column on the Nodes sheet. Unified format is current for InfiniBand, Ethernet, and NVLink and does not use a DNS toggle or separate IP-mapping file.

Legacy IB P2P, .topo, and .dot files remain supported. For these formats, DNS resolution is enabled by default. If device names are not resolvable, disable DNS and provide an IP mapping file.

Review the Unified template and file structure in https://networking-docs.nvidia.com/cablevalidationtool/2.0.1/p2p-file. After loading, verify the switch, host, and link counts in the topology summary. A different topology cannot be loaded while validation is running.

Step 4 - Set Credentials

In Credentials, configure the accounts CVT will use to access the switches. Switches require a password. Default credentials apply to switches without a node-specific account; per-node credentials and credential profiles override the defaults.

Credentials must be configured before pre-check and agent deployment. See https://networking-docs.nvidia.com/cablevalidationtool/2.0.1/agent-deployment-and-ssh-configuration.

Step 5 - Run Pre-check and Deploy Agents

Pre-check

In Pre-Check, run readiness checks for:

  • Ping

  • Management API

  • SSH

  • Docker

  • LLDP

  • Existing agent status

Fix failed checks before deployment. A switch that cannot be reached remains outside validation until it is connected to the management network.

Deploy Agents

In Deploy Agents, select Deploy Agents on all devices in the topology to cover every eligible switch. To deploy selectively, apply a filter and move the required devices into the selected list.

CVT deploys only to devices marked Agent Ready. Devices that failed pre-check remain Not Provisioned and are skipped. Keep Force deploy disabled; it bypasses the status check. Deployment is unavailable while validation is running.

Device progression is generally Undefined → Provisioned → Agent Ready → Validation Ready. Agent status and removal are managed from https://networking-docs.nvidia.com/cablevalidationtool/2.0.1/agent-management.

Host agents are controlled differently depending on the topology format. In a Unified workbook, there is no Manage Servers checkbox; a host gets an agent when its Managed cell is set to yes, and that cell defaults to yes. The Manage Servers checkbox exists only for Legacy Ethernet and DOT loads, where it is off by default.

Step 6 - Start Validation

In Validate, review the setup summary and device readiness, then select Start Validation. A progress window displays live results and can be minimized while validation continues.

Stop validation from the same step or from System Admin → Collector Management → Service Controls when the pass is complete.

Step 7 - Review Results and Remediate

Use the Cable Validation UI as the record for the validation pass.

  • Circuits: Shows each planned connection, its Pass / Fail / Incomplete state, and remediation history. The default view lists failing circuits.

  • Agent Management: Shows which nodes are reporting.

  • Syndrome details: Use the CVT introduction to interpret wiring, link-integrity, and provisioning findings.

Typical Findings

Category

Examples

Wiring

Wrong neighbor, wrong port, unknown neighbor, extra cable, or unplugged media.

Link integrity

Link down with no signal, flapping, underperforming BER, anomalous port, or RX/TX power mismatch.

Provisioning

Unreachable device, administrative shutdown, negotiation failure, or missing report.

Confirm a deployed portion: All connections expected to be installed are Pass, and any remaining failures are explained by equipment or media that is not yet installed.

Repeat Validation as Deployment Progresses

Validation can be repeated as additional switches and links become available. For each subsequent pass:

  1. Run Pre-Check for newly reachable devices.

  2. Deploy agents to devices that become Agent Ready.

  3. Start validation again.

  4. Review the newly introduced failures in Circuits and remediate them.

Reference Documentation

Last updated: