OpenSM Static Topology Configuration REST API

Alongside the dynamic discovery of the InfiniBand network, OpenSM can also function with a static network configuration. This static configuration is specified in a configuration file topoconfig.conf, which is by default located at /opt/ufm/files/conf/opensm/topoconfig.conf.

To enable OpenSM to operate with static network configurations, you need to set the appropriate mode in the OpenSM configuration file located at /opt/ufm/files/conf/opensm/opensm.conf.

When UFM is set to use static configuration (as defined in the gv.cfg file), OpenSM will be configured accordingly during the UFM service startup.

The static configuration file (topoconfig.conf) defines the fabric links, and any other links present in the fabric will not be discovered or included in the OpenSM model.

Topoconfig File

The format of the topoconfig.conf file is as follows:

0x98039b0300867bba,1,0xb83fd2030080302e,1,Any,Active
0x98039b0300867bba,3,0xb83fd2030080302e,3,Any,Active
0xb83fd2030080302e,1,0x98039b0300867bba,1,Any,Active
0xb83fd2030080302e,3,0x98039b0300867bba,3,Any,Active
0xb83fd2030080302e,26,0xf452140300280040,1,Any,Activ
0xb83fd2030080302e,29,0xf452140300280080,1,Any,Active
0xb83fd2030080302e,30,0xf452140300280081,1,Any,Active

Header:

Source GUID
Source Port
Destination GUID
Destination Port
Device Type
Link State

The header is for understanding the field order and meaning but is not part of the file itself.

UFM Configuration to Enable Static Discovery by OpenSM

To allow UFM and/or OpenSM to utilize the static topology configuration, set the following parameter in the gv.cfg file:

[SubnetManager]
... ... .. ... ..
# This parameter defines if topoconfig file could be used for opensm discovery.
topoconfig_enabled = True

After making this change, restart UFM using the command: /etc/init.d/ufmd restart.

If configured for static topology, the OpenSM configuration files will be updated during the UFM startup process:

# The file holding the topo configuration.
topo_config_file (null)
# If set to true, the SM will adjust its operational
# mode to consider the topo_config file.
topo_config_enabled FALSE

Will be modified to:

# The file holding the topo configuration.
topo_config_file /opt/ufm/files/conf/opensm/topoconfig.conf.
# If set to true, the SM will adjust its operational
# mode to consider the topo_config file.
topo_config_enabled TRUE

If the topoconfig file does not exist or is empty, OpenSM will revert to dynamic discovery.

REST APIs to Manage Topoconfig File

Create SM Topoconfig File Based on Current Topology

This option allows for the creation of a topoconfig file based on the current topology, useful for minimizing disruption when adding new equipment. The file can be edited manually if necessary and then deployed to UFM.

Request URL:

/ufmRest/static_topology/sm_topology_file
Method:

POST
Status Codes: 202 – Accepted

Example:

# curl  -k -u admin:123456 -X POST  https:// [UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topology_file

This action is asynchronous, and the response will include a job ID.

Response:

{
    "SM topoconfig action": "Create topoconfig file",
    "job_id": "1"
 }

You can check the job status using the provided job ID.

https:// [UFM_Server_name_or_IP ]/ufmRest/jobs/[job id]
curl  -k -u admin:123456 -X GET https:// UFM_Server_name_or_IP /ufmRest/jobs/1

Response:

{
    "ID": "1",
    "Status": "Completed",
    "Progress": 100,
    "Description": "Create opensm topoconfig file",
    "Created": "2024-09-27 08:14:32",
    "LastUpdated": "2024-09-27 08:14:32",
    "Summary": "/tmp/ibdiagnet_out/generated_topoconfig.conf",
    "RelatedObjects": "",
    "CreatedBy": "admin",
    "Operation": "opensm topoconfig file management",
    "Foreground": true,
    "SiteName": null
}

Once the job is completed, the path to the generated topoconfig file on the UFM server will be included in the job completion message (Summary).

The default location for the generated topoconfig file is:

/tmp/ibdiagnet_out/generated_topoconfig.conf

Deploy Topoconfig File to UFM

This option allows you to deploy topoconfig file to UFM (OpenSM). The topoconfig file could be created manually or using the above REST API.

Request URL:

/ufmRest/static_topology/master_topoconfig
Method:

POST
Status Codes: 200 – OK
Example:
```
curl -k -u admin:123456 -X POST -F file=@/config/topoconfig/topoconfig_test.csv https://[UFM_Server_name_or_IP]/ufmRest/static_topology/master_topoconfig
```
UFM will deploy the topoconfig file and trigger OpenSM to reload its configuration. This process is synchronous, and the status will indicate success or failure.

Get Current Topoconfig File from UFM

Retrieves the currently deployed topoconfig file from UFM. The file can be analyzed, modified if necessary, and redeployed to UFM.

Request URL:

/ufmRest/static_topology/sm_topology_file
Method:

GET
Status Codes: 200 – OK

Example:

curl -k -u admin:123456 -X GET https://[UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topology_file

This will return the current OpenSM topoconfig file if it exists.

Response:

0x98039b0300867bba,1,0xb83fd2030080302e,1,Any,Active                                                                                                                                                                                         
0x98039b0300867bba,3,0xb83fd2030080302e,3,Any,Active                                                                                                                                                                                         
0x98039b0300867bba,11,0xc42a1030002227a,1,Any,Active
0x98039b0300867bba,39,0xc42a103000a6662,1,Any,Active
0xb83fd2030080302e,1,0x98039b0300867bba,1,Any,Active
0xb83fd2030080302e,3,0x98039b0300867bba,3,Any,Active
0xb83fd2030080302e,26,0xf452140300280040,1,Any,Active
0xb83fd2030080302e,29,0xf452140300444080,1,Any,Active
0xb83fd2030080302e,30,0xf452144400280081,1,Any,Active

Create Full Diff Report Between Current SM Topoconfig File and Current Fabric Configuration

If there have been hardware changes in IB fabric, you can view the differences between the current network state and the definitions in the topoconfig file. All the changes, related to described in current topoconfig file links, will be included in report: New devices, new links, removed devices, changed links that are not part of current topoconfig file will be in the report.

Request URL:

/ufmRest/static_topology/sm_topo_diff_report
Method:

POST
Status Codes: 202 – Accepted

Example:

curl -k -u admin:123456 -X POST https://[UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topo_diff_report

The operation is asynchronous, and the response will include a job ID. You can check the job status using the job ID.

The diff is generated from the topoconfig file, displaying only the links that have changed from the original file.

presented in the report. New links or deleted links will not be shown in report.

{
    "SM topoconfig action": "Create topoconfig diff report.",
    "job_id": "2"
}

Check for job status using:

https:// [UFM_Server_name_or_IP ]/ufmRest/jobs/[job ID]
(curl  -k -u admin:123456 -X GET  https:// [UFM_Server_name_or_IP]/ufmRest/jobs/[job ID])
{
    "ID": "2",
    "Status": "Completed",
    "Progress": 100,
    "Description": "Report opensm topoconfig file diff",
    "Created": "2024-09-29 06:02:10",
    "LastUpdated": "2024-09-29 06:02:10",
    "Summary": "/opt/ufm/files/reports/TopologyCompare/sm_topo_diff_report.json",
    "RelatedObjects": "",
    "CreatedBy": "admin",
    "Operation": "opensm topoconfig file management",
    "Foreground": true,
    "SiteName": null
}

Create Diff Report Between Current SM Topoconfig File and Current Fabric Configuration for replaced HCAs

If there have been hardware changes, related to replacement of host HCAs, you can view the differences between the current network state and the definitions in the topoconfig file. Only changes, related to described in current topoconfig file links, will be included in report. New devices or links that are not part of current topoconfig file will not be part of report.

Request URL:

/ufmRest/static_topology/sm_topo_diff_report?hca_only=true
Method:

POST
Status Codes: 202 – Accepted

Example:

curl -k -u admin:123456 -X POST https://[UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topo_diff_report?hca_only=true

The operation is asynchronous, and the response will include a job ID. You can check the job status using the job ID.

The diff is generated from the topoconfig file, displaying only the links that have changed from the original file.

presented in the report. New links or deleted links will not be shown in report.

{
    "SM topoconfig action": "Create topoconfig diff report.",
    "job_id": "2"
}

Check for job status using:

https:// [UFM_Server_name_or_IP ]/ufmRest/jobs/[job ID]
(curl  -k -u admin:123456 -X GET  https:// [UFM_Server_name_or_IP]/ufmRest/jobs/[job ID])
{
    "ID": "3",
    "Status": "Completed",
    "Progress": 100,
    "Description": "Report opensm topoconfig file diff",
    "Created": "2024-09-29 06:05:17",
    "LastUpdated": "2024-09-29 06:05:20",
    "Summary": "/opt/ufm/files/reports/TopologyCompare/sm_topo_diff_report.json",
    "RelatedObjects": "",
    "CreatedBy": "admin",
    "Operation": "opensm topoconfig file management",
    "Foreground": true,
    "SiteName": null
}

Show SM Topology Diff Report

To view the report created by the previous API call, request the topodiff report, which shows the differences between the topoconfig file and the current fabric links. Differences will be shown only for links described in topoconfig file - in case of HCAs replacement. New links, discovered in fabric - as a result of additions of new devices, will not be shown.

Request URL:

/ufmRest/static_topology/sm_topo_diff_report
Method:

GET
Status Codes: 200 – OK
Example:
```
curl -k -u admin:123456 -X GET https://[UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topo_diff_report
```
This action is synchronous and will return a report detailing all changed links with updated GUIDs and port numbers. The report will be shown according to the lastly created report: full fabric or only for replaced HCA.

Response for full report:

{
    "Added Links": {
        "1": {
            "Added Link": "GUID 0xb83fd2030080302e (MF0;r-ufm-sw13:MQM8700/U1) Port 26 to GUID 0xf452140300280040 (r-ufm64 HCA-1) Port 1"
        }
    },
    "Removed Links": {
        "1": {
            "Removed Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 15 to GUID 0xc42a103000a2e7b (Unknown) Port 1"
        },
        "2": {
            "Removed Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 9 to GUID 0xb83fd2030080302e (r-ufm-sw13) Port 9"
        },
    },
    "Changed Links": {
        "1": {
            "Changed Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 11 to GUID 0xc42a10300092e7a (Unknown) Port 1 peer changed to GUID 0xc42a103000a2e7a (r-ufm57 mlx5_0) Port 1"
        }
    }
}

Response for report with key hca_only=true - only for replaced HCAs:

{
    "1": {
        "Topoconfig Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 11 to GUID 0xc42a1030002227a Port 1",
        "Current Topology Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 11 to GUID 0xc42a103000a2e7a Port 1 (r-ufm57)"
    },
    "2": {
        "Topoconfig Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 39 to GUID 0xc42a103000a6662 Port 1",
        "Current Topology Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 39 to GUID 0xc42a103000a2e62 Port 1 (r-ufm142)"
    },
    "3": {
        "Topoconfig Link": "GUID 0xb83fd2030080302e (r-ufm-sw13) Port 29 to GUID 0xf452140300444080 Port 1",
        "Current Topology Link": "GUID 0xb83fd2030080302e (r-ufm-sw13) Port 29 to GUID 0xf452140300280080 Port 1 (r-ufm55)"
    },
    "4": {
        "Topoconfig Link": "GUID 0xb83fd2030080302e (r-ufm-sw13) Port 30 to GUID 0xf452144400280081 Port 1",
        "Current Topology Link": "GUID 0xb83fd2030080302e (r-ufm-sw13) Port 30 to GUID 0xf452140300280081 Port 1 (r-ufm55 mlx5_1)"
    }
}
In report possible to see all the changed links with updated GUIDs and port numbers for replaces HCAs.
Initial definition in topoconfig file
"Topoconfig Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 39 to GUID 0xc42a103000a6662 Port 1",

Current link parameters (not currently detected by OpenSM, as they are not defined in the topoconfig file):

"Current Topology Link": "GUID 0x98039b0300867bba (NEMO-LEAF-2) Port 39 to GUID 0xc42a103000a2e62 Port 1 (r-ufm142)"

Patch/Update Original Topoconfig File with Changes in Current Fabric

If hardware has been replaced, the original topoconfig file can be updated accordingly, and OpenSM will be signaled to reread its configuration.

Request URL:

/ufmRest/static_topology/sm_topology_file
Method:

PATCH
Status Codes: 202 – Accepted

Example:

curl -k -u admin:123456 -X PATCH https://[UFM_Server_name_or_IP]/ufmRest/static_topology/sm_topology_file

Response:

urls=/ufmRest/static_topology /sm_topology_file
methods="PATCH"

{
    "SM topoconfig action": "Update topoconfig file",
    "job_id": "4"
}
 
Check job status using
https:// [UFM_Server_name_or_IP ]/ufmRest/jobs/[job ID]
(curl  -k -u admin:123456 -X GET  https:// [UFM_Server_name_or_IP]/ufmRest/jobs/[job ID])
 
 Respond:
{
    "ID": "4",
    "Status": "Completed",
    "Progress": 100,
    "Description": "Update opensm topoconfig file",
    "Created": "2024-09-29 06:18:26",
    "LastUpdated": "2024-09-29 06:18:27",
    "Summary": "/opt/ufm/files/conf/opensm/topoconfig.conf updated",
    "RelatedObjects": "",
    "CreatedBy": "admin",
    "Operation": "opensm topoconfig file management",
    "Foreground": true,
    "SiteName": null
}

The response will include a job ID to track the operation status.

Once completed, OpenSM will reload the fabric based on the updated topoconfig file.

Last updated: January 22, 2025