OpenSSL and PKA | BlueField Troubleshooting Guide

Preface

OpenSSL library is used by applications to handle security communication like TLS/HTTPS along with certificate/key creation and management. BlueField enables high-performance computing by offloading requests to PKA hardware acceleration. The PKA library implements an OpenSSL engine to interact with the OpenSSL library and the PKA hardware.

The following are the layers of components from the application that interacts with the API/OpenSSL library down to the hardware: Application > OpenSSL Library > PKA Library > PKA driver > PKA hardware.

Command Cheat Sheet

Command	Description
`lscpu`	Linux command to display CPU information
`lsmod`	Linux command to list loaded kernel modules
`pka_test_validation`	BlueField test program to verify PKA library and hardware in multithreaded scenario
`openssl`	OpenSSL command to interact with OpenSSL library. Provides certificate/key creation, testing and verification

Logging and Counters

Logging is managed by applications using OpenSSL and PKA libraries, similar to Nginx or Apache web servers. You can find the log files in /var/log/nginx or /var/log/access_log.

OpenSSL offers APIs for tracing, which can be found at SSL_CTX_set_msg_callback. Applications should utilize these function calls to trace SSL/TLS protocols.

The PKA API functions communicate status and error information back to the OpenSSL library, which logs these messages if configured to do so. These messages are sent to stderr and depend on the application to redirect them to the application log.

Currently, counters are not exposed to users and are intended for development purposes only.

As an alternative to file logging, using packet capture tools like tcpdump can provide detailed information about the TLS handshake, including status and errors, which is often invaluable for debugging.

Debug Info Package

N/A

Scenarios

Common issues related to OpenSSL and PKA:

PKA Engine Loaded Incorrectly

Library Missing Error

The following indicates that the location for PKA engine shared library file is missing.

$ openssl engine pka
20B095A0FFFF0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:../crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/aarch64-linux-gnu/engines-3/pka.so): /usr/lib/aarch64-linux-gnu/engines-3/pka.so: cannot open shared object file: No such file or directory

Version Mismatch Error

The following indicates that the libPKA.so.1 shared library cannot find a definition of function ASYNC_WAIT_CTX_get_fd().

$ openssl speed -engine pka -async_jobs 8 rsa
Error configuring OpenSSL
281474842441520:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(/usr/lib64/openssl/engines/libpka.so): /lib64/libPKA.so.1: undefined symbol: ASYNC_WAIT_CTX_get_fd

This is part of the OpenSSL 3.0 async_jobs feature. OpenSSL's shared library should provide this function definition.

Insufficient PKA Resources

PKA hardware has limited PKA ring resources. Bluefield 2 has 32 rings, Bluefield 3 MB has 64 rings, and Bluefield 3 HB has 96. Each process invokes opensll and pka engine will take 4 rings(default) during the lifetime of the process. When pka resource is exhausted. The pka engine will not be loaded correctly.

 $ openssl engine pka
(pka) PKA handle is invalid

Known OpenSSL Issue

The following message is benign and can be ignored:

$ openssl engine pka
(pka) BlueField PKA engine support
2090E585FFFF0000:error:1280006A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:../crypto/dso/dso_dlfcn.c:188:symname(EVP_PKEY_base_id): /usr/lib/aarch64-linux-gnu/engines-3/pka.so: undefined symbol: EVP_PKEY_base_id
2090E585FFFF0000:error:1280006A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:../crypto/dso/dso_lib.c:176:

OpenSSL Utility Cannot Create Certificates

In rare cases, some crypto algorithms are not supported by the OpenSSL library due to compile time configuration. If you run into such an error, please report it to NVIDIA Enterprise Support.

Application Failure due to Error Returned by OpenSSL Function Calls

Example of this is the TLS/OpenSSL connection handshake. The Wireshark or tcpdump handshake will have the error code.

The following is an example response from the HTTPS/TLS server to TLS Client Hello request as captured by Wireshark:

TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
    Content Type: Alert (21)
    Version: TLS 1.2 (0x0303)
    Length: 2
    Alert Message
        Level: Fatal (2)
        Description: Handshake Failure (40)

Possible TLS/HTTPS handshaking errors

Network related issue: interface provision, wiring, etc...
TLS errors: TLS versions between client and server, no matching cipher suites/supporting groups/signature algorithms/key management etc...

Please report this error with both information of both client and server-side information. Such as wget for HTTPS client, Apache webserver or google.com as HTTS server.

For all other issues, please follow the below steps to narrow down the errors and report it.

Diagnosing Issues with PKA Library

Make sure CPU support AES algorithm:

$ lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A78AE
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 8
    Socket(s):          1
    Stepping:           r0p1
    BogoMIPS:           600.50
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

Check if PKA kernel module is loaded and devices(pka rings) are functioning.

$ lsmod | grep pka
mlxbf_pka             126976  0

$ ls /dev/pka/
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  8
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  7   9

Run PKA validation test program:

$ /usr/bin/pka_test_validation -c 8 -r 4 -s 1
PKA system info
---------------
PKA API version: v1
Cache line size: 64
CPU count:       8
Ring count:      4


Running PKA inst: pka_test_validation
-----------------
Avail rings:      4
HW rings in use      :  000000000001111

Mode:            PKA_F_PROCESS_MODE_SINGLE(bit 1)
Sync:            PKA_F_SYNC_MODE_ENABLE(bit 8)


num worker threads: 8
Starting thread_idx=0 on cpu_num=0
Starting thread_idx=1 on cpu_num=1
Starting thread_idx=2 on cpu_num=2
Starting thread_idx=3 on cpu_num=3
Starting thread_idx=4 on cpu_num=4
Starting thread_idx=6 on cpu_num=6
Starting thread_idx=7 on cpu_num=7
Starting thread_idx=5 on cpu_num=5
thread_start_routine thread_idx=0 cpu_num=0 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=1 cpu_num=1 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=2 cpu_num=2 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=3 cpu_num=3 done
        tests_passed=6
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=4 cpu_num=4 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=5 cpu_num=5 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=6 cpu_num=6 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=7 cpu_num=7 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
validation tests passed!

Diagnosing Issues with OpenSSL

Verify OpenSSl version

$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Verify if PKA is loaded by Openssl as default. News bfb would NOT have PKA loaded by default.
```
$ openssl engine
(dynamic) Dynamic engine loading support
```
Check if OpenSSL can load PKA engine/library.
```
$ openssl engine pka
(pka) BlueField PKA engine support
```
The printed errors are benign and can be ignored for now.

Check if OpenSSL speed test can load PKA and offload requests to PKA engine in SYNC mode.

$ openssl speed -engine pka rsa
Engine "pka" set.
Doing 512 bits private rsa's for 10s: 131665 512 bits private RSA's in 9.97s
Doing 512 bits public rsa's for 10s: 556640 512 bits public RSA's in 10.00s
Doing 1024 bits private rsa's for 10s: 38463 1024 bits private RSA's in 9.99s
...
PKA_ENGINE: pka_modular_exp_crt failed, rc =-1504 Key length reaches PKA hardware limitation
RSA sign setup failure.  No RSA sign will be done.
2040969CFFFF0000:error:1C880004:Provider routines:rsa_sign:RSA lib:../providers/implementations/signature/rsa_sig.c:652:
RSA verify setup failure.  No RSA verify will be done.
version: 3.0.2
built on: Wed Jan 31 18:43:23 2024 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-zqLXWk/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbf
                  sign    verify    sign/s verify/s
rsa  512 bits 0.000076s 0.000018s  13206.1  55664.0
rsa 1024 bits 0.000260s 0.000033s   3850.2  30368.3
rsa 2048 bits 0.001435s 0.000087s    696.9  11498.8
rsa 3072 bits 0.004575s 0.000173s    218.6   5764.5
rsa 4096 bits 0.010101s 0.000274s     99.0   3651.7

PKA library currently supports only up to 4096 bits RSA.

Check if OpenSSL speed test can load PKA and offload requests to PKA engine in ASYNC mode with async_jobs.

$ openssl speed -engine pka -async_jobs 8 dsa
Engine "pka" set.
Doing 512 bits sign dsa's for 10s: 152074 512 bits DSA signs in 8.05s
Doing 512 bits verify dsa's for 10s: 173404 512 bits DSA verify in 7.04s
Doing 1024 bits sign dsa's for 10s: 142377 1024 bits DSA signs in 8.04s
Doing 1024 bits verify dsa's for 10s: 148043 1024 bits DSA verify in 7.08s
Doing 2048 bits sign dsa's for 10s: 116583 2048 bits DSA signs in 7.44s
Doing 2048 bits verify dsa's for 10s: 68874 2048 bits DSA verify in 4.79s
version: 3.0.2
built on: Wed Jan 31 18:43:23 2024 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-zqLXWk/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbf
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000053s 0.000041s  18891.2  24631.2
dsa 1024 bits 0.000056s 0.000048s  17708.6  20910.0
dsa 2048 bits 0.000064s 0.000070s  15669.8  14378.7

Due to an OpenSSL speed test issue with async_jobs and RSA, DSA algorithm is used here.

PKA library currently support only up to 4096 bits RSA.

Last updated: September 18, 2024