Creating a Topology File

Topology File Generation

The topology file describes the physical connections between all cluster elements (switches, hosts, HCAs). It serves three purposes:

• Documentation — standardized record of the planned topology

• Validation baseline — serves as the reference for verifying deployed cabling. To confirm that the physical fabric matches the plan, the topology file must be compared against a live fabric discovery (e.g. using ibtopodiff against ibdiagnet2.lst or ibnetdiscover output). The topology file alone does not perform this verification.

• Troubleshooting — maps and visualizes the topology when diagnosing issues

Scope: This procedure applies only to NDR (MQM9700 / ConnectX-7). For XDR (Q3400/ ConnectX-8) clusters, please refer to the Confirming Topology - XDR chapter in this document.

Prerequisites

Extract the parser:

tar xzf IB_Topology_files_data_for_RD.tgz

Understanding Port Identities

Three different port identities are used throughout this process. Do not confuse them:

The hcaPortMapping dictionary in the parser script translates PTP logical labels → Linux interface names. The parser then maps these to the IBNL model ports in the generated topology file.

Step 1 — Verify the HCA Port Mapping

Inside parse_ptp_file.py (at the top of the source code), there is a dictionary named hcaPortMapping. It maps each PTP logical label to the corresponding Linux interface name.

hcaPortMapping = {
    'U1/P1': 'mlx5_3/P1',
    'U1/P2': 'mlx5_1/P1'
}

This means:

• PTP label U1/P1 → Linux interface mlx5_3, port 1

• PTP label U1/P2 → Linux interface mlx5_1, port 1

Verify the mapping for each server hardware profile — not just one representative node, unless all nodes in the cluster are physically identical in HCA placement and wiring.

For each hardware profile:

1. SSH into a node of that profile.

2. Identify each HCA by multiple attributes (OS name alone is not sufficient):

   Bash

   # HCA GUID, state, and port info
   ibstat
   
   # PCI location → interface mapping
   ls -l /sys/class/infiniband/*/device | awk -F/ '{print $(NF-2), $NF}'
   
   # Verify which switch port each local HCA port is connected to
   iblinkinfo | grep $(hostname)
   
   # Or from ibdiagnet output:
   grep $(hostname) ibdiagnet2.lst
   
   

3. For each PTP label (U1/P1, U1/P2, …), confirm:

   • The correct Linux interface (mlx5_N)

   • The HCA GUID matches expectations

   • The PCI slot matches the physical HCA location

   • The connected switch peer port (from iblinkinfo or ibnetdiscover) matches the PTP plan

4. Update the dictionary for any hardware profile that differs.

Step 2 — Pre-Generation Mapping Checks

Before running the parser, confirm the mapping passes these gates.

Quick self-check:

python3 -c "
from parse_ptp_file import hcaPortMapping
labels = list(hcaPortMapping.keys())
targets = list(hcaPortMapping.values())
print(f'Labels: {len(labels)}, Unique targets: {len(set(targets))}')
assert len(labels) == len(set(targets)), 'ERROR: duplicate target mapping detected'
print('Mapping OK - no duplicates')
"

Step 3 — Run the Parser Script

From the directory containing the script, IBNL model files, and PTP workbook:

python3 parse_ptp_file.py parse -f <ptp-workbook>.xls -of -mhp

Expected output:

• output.topo — the generated topology file

• ibdmchk validation output (run automatically by the script)

Step 4 — Validate ibdmchk Results

Record the exact command (run by the script, or run manually):

ibdmchk -t output.topo

Acceptance criteria:

Save the full ibdmchk output (stdout + stderr), it is a required archival artifact.

Step 5 — Post-Generation Validation Checks

5a — Authoritative Count Comparison

Compare the generated topology against the approved PTP workbook (the single source of truth for expected counts):

# Count unique switches (system header lines)
grep -cP '^\S+.*CFG' output.topo

# Count unique hosts
grep -oP '(?<=-> )\S+ \S+' output.topo | awk '{print $2}' | sort -u | wc -l

# Count total links (endpoint pairs)
grep -c "\->" output.topo

5b — Structural Checks

# Check for duplicate endpoint pairs (should return nothing)
grep "\->" output.topo | awk '{print $1, $NF}' | sort | uniq -d

# List all unique system names for review
grep -oP '^\S+|(?<=-> )\S+ \S+' output.topo | sort -u

Step 6 — Rename and Archive

Rename the file with a meaningful convention:

mv output.topo <site>_<cluster>_topology_<YYYY-MM-DD>.topo

Mandatory archival checklist — store all items together in a shared, reachable location:

# Generate SHA-256 hashes
sha256sum <site>_<cluster>_topology_<YYYY-MM-DD>.topo <ptp-workbook>.xls > checksums.sha256

This topology file is used in subsequent stages including cable validation, fabric diagnostics, and maintenance verification.

Topology File Example

The following is an illustrative example. Exact system names, port identities, and link speeds depend on your platform, IBNL model, and hcaPortMapping. Replace with actual validated output from your generation run.

MQM9700 cl02s0lclf01 CFG : main=4x
    P1 -4x-100G-> HCA_mlx5_3 cl02s0ldgx01 P1
    P2 -4x-100G-> HCA_mlx5_3 cl02s0ldgx02 P1
    P3 -4x-100G-> HCA_mlx5_3 cl02s0ldgx03 P1
    P4 -4x-100G-> HCA_mlx5_3 cl02s0ldgx04 P1
    ...

Saving the Topology File

After creating the Point-to-Point excel file and the topology file, make sure to save the file for future usage. 

Facilitating the topology enables you to apply the master topology settings to the current topology, when the final topology is reached. This ensures that the fabric is in a consistent and optimal state. The master topology is a reference point that represents the desired state of the fabric. It can be set by selecting the latest topology or by uploading a predefined custom topology from a file.
Periodic comparison allows users to compare the current fabric topology with a preset master topology. By comparing the current topology with the master topology, users can detect any deviations, errors, or anomalies in the fabric and take corrective actions if needed.

Last updated: October 22, 2026