NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance
NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance
Breadcrumbs

APPENDIX—Skyway Multi-PKEY Configuration With Traffic Isolation

This apprendix describes a way to configure the InfiniBand cluster connected to Skyway to support traffic isolation per HCA/host.


The “ALL” option will configure all the HCA GUIDs in the fabric, including the vGUIDs for both the default and non-default PKEYs.

Example of OpenSM partitions.conf file

cat /etc/opensm/partitions.conf 
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;


The traffic per host needs to be isolated (i.e., all HCA GUIDs should not be members of the non-default partitions). Therefore, the actual required HCA GUIDs need to be added to the non-default partition and included the 8 Skyway GUIDs.

Example:

Test=0x1, ipoib, defmember=full :  ALL_SWITCHES=full,SELF=full; 0x…..1, 0x……2, skyway guid1, skyway guid2, …., skyway guid8;

 
All Skyway GUIDs including the vGUIDs are needed: 

  1. Configure the partitions.conf file on the SM node.

    cat /etc/opensm/partitions.conf 
Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;


  2. Run ibdiagnet on one of the InfiniBand host.

  3. Get all Skyway GUIDs from the ibdiagnet2.vports_pkey file: 

    # cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep <skyway host name> | awk -F "guid=" '{print $2}' | paste -s -d "," | sed -e 's/,/, /g' > filename

    Lab example:

    # cat /var/tmp/ibdiagnet2/ibdiagnet2.vports_pkey | grep skyway | awk -F "guid=" '{print $2}' | paste -s -d "," | sed -e 's/,/, /g' > /tmp/skyway_guids.txt


  4. Edit this file to add the needed partitions.conf configuration and replace it with the old one.

Example of partitions.conf File 

Default=0x7fff, ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Test=0x1, ipoib, defmember=full : ipoib, defmember=full : ALL_SWITCHES, SELF, 0x58a2e10300470a66,0x58a2e103004709ca, 0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260

Example of skyway_guids.txt File 

0x001405000a799403, 0x001405000a799405, 0x001405000a799404, 0x001405000a799402, 0x001405000a799401, 0x0c42a1030098b250, 0x001405000a799406, 0x001405000a799407, 0x0c42a1030098b2b8, 0x001405000a799439, 0x001405000a79943a, 0x001405000a79943b, 0x001405000a79943c, 0x001405000a79943d, 0x001405000a79943e, 0x001405000a79943f, 0x001405000a799412, 0x001405000a799413, 0x001405000a799414, 0x001405000a799415, 0x001405000a799416, 0x001405000a799429, 0x001405000a79942a, 0x001405000a79942b, 0x001405000a799409, 0x001405000a79940a, 0x001405000a79940c, 0x0c42a1030098b230, 0x001405000a799411, 0x001405000a79941a, 0x001405000a79942d, 0x001405000a799433, 0x001405000a799435, 0x001405000a79940e, 0x001405000a79940f, 0x001405000a799418, 0x001405000a799427, 0x001405000a79942e, 0x001405000a79942f, 0x001405000a799430, 0x001405000a799431, 0x001405000a799408, 0x001405000a79940d, 0x001405000a799417, 0x001405000a799419, 0x001405000a79941b, 0x001405000a799428, 0x001405000a799436, 0x001405000a799437, 0x001405000a79940b, 0x001405000a799410, 0x001405000a799424, 0x001405000a799425, 0x001405000a799426, 0x001405000a79942c, 0x001405000a799432, 0x001405000a799434, 0x001405000a79941c, 0x001405000a79941d, 0x001405000a79941e, 0x001405000a79941f, 0x001405000a799420, 0x001405000a799421, 0x001405000a799422, 0x001405000a799423, 0x0c42a1030098b228, 0x0c42a1030098b290, 0x001405000a799438, 0x001405000a799400, 0x0c42a1030098b270, 0x0c42a1030098b268, 0x0c42a1030098b260


Last updated: