OpenSsl, tcnative with pkcs#11 hsm

[Andreas Grunz]

I want to use Netty together with OpenSsl and a hardware security module (hsm). As a proof of concept I succesfully started a OpenSsl server connected to a hsm by using the following command and configuration.

env OPENSSL_CONF=openssl.cnf openssl s_server -curves brainpoolP256r1 -engine pkcs11 -keyform engine -cert TLS-Cert.pem -key 'pkcs11:object=f4ec939ae6ddbd924b3c24893630e1291be92984'

Content of openssl.cnf

openssl_conf = openssl_def

[openssl_def]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/pkcs11.so
# HSM lib
MODULE_PATH = /mylibs/libcs_pkcs11_R3.so
PIN = "0000"

Now i am struggling to transfer this to netty with tcnative. We compiled tcnative with OpenSsl static. A tls handshake with Netty, OpenSsl and without a hsm is possible by the using the static library.

SslContext sslCtx = SslContextBuilder
                .forServer(
                        EchoServer.class.getResourceAsStream("/TLS_Pub.pem"),
                        EchoServer.class.getResourceAsStream("/TLS-Key.pem"))
                .protocols(SslProtocols.TLS_v1_2)
                .ciphers(ciphers, IdentityCipherSuiteFilter.INSTANCE_DEFAULTING_TO_SUPPORTED_CIPHERS)
                .sslProvider(SslProvider.OPENSSL)
                .build();

Now I wondering how to add the hsm support?

I tried to use

File file = new File("/tmp/keys/TLS_Pub.crt");
OpenSslX509KeyManagerFactory factory =  OpenSslX509KeyManagerFactory.newEngineBased(file, null);

 SslContext sslCtx = SslContextBuilder
                .forServer(factory)
                .protocols(SslProtocols.TLS_v1_2)
                .ciphers(ciphers, IdentityCipherSuiteFilter.INSTANCE_DEFAULTING_TO_SUPPORTED_CIPHERS)
                .sslProvider(SslProvider.OPENSSL)
                .build();

but i get the following exception

Exception in thread "main" java.security.UnrecoverableKeyException: Unable to load key from engine
      at io.netty.handler.ssl.OpenSslX509KeyManagerFactory$OpenSslKeyStore$1.engineGetKey(OpenSslX509KeyManagerFactory.java:316)
      at java.base/java.security.KeyStore.getKey(Unknown Source)
      at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(Unknown Source)
      at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(Unknown Source)
      at java.base/javax.net.ssl.KeyManagerFactory.init(Unknown Source)
      at io.netty.handler.ssl.OpenSslX509KeyManagerFactory$OpenSslKeyManagerFactorySpi.engineInit(OpenSslX509KeyManagerFactory.java:128)
      at java.base/javax.net.ssl.KeyManagerFactory.init(Unknown Source)
      at io.netty.handler.ssl.OpenSslX509KeyManagerFactory.newEngineBased(OpenSslX509KeyManagerFactory.java:262)
      at io.netty.handler.ssl.OpenSslX509KeyManagerFactory.newEngineBased(OpenSslX509KeyManagerFactory.java:246)
      at netty.testing.echoserver.EchoServer.main(EchoServer.java:40)
Caused by: java.lang.Exception: Unable to load private key (error:26096043:engine routines:ENGINE_load_private_key:passed a null parameter)
      at io.netty.internal.tcnative.SSL.loadPrivateKeyFromEngine(Native Method)
      at io.netty.handler.ssl.OpenSslX509KeyManagerFactory$OpenSslKeyStore$1.engineGetKey(OpenSslX509KeyManagerFactory.java:313)

Netty is using "key" as alias. I am wondering how one can create a delegate key or alias / label configuration to access a specific PKCS#11 key.

[ramtech123]

@grun077 could you please share some details on how did you resolve this issue?

[Andreas Grunz]

To set the alias i implemented two approaches. I implemented a custom keymanager:

CustomServerAliasKeyManager extends X509ExtendedKeyManager

and i tried a custom keystore

CustomServerAliasKeyStore extends KeyStore

In both cases I can set the key alias as a parameter in the constructor.

During debugging I can see that the right alias is used and passed. Unfortunately i was not able to retrieve the private key. I got a internal openssl error.