Remote jmx has enabled SSL. How to configure jmx-exporter for verification?

Question

Remote jmx has enabled SSL. How to configure jmx-exporter for verification?

kingEneru opened this issue 4 months ago · comments

issue description

jmx_exporter able to running successful in kubernetes, but remote jmx has enabled SSL. So, when I access to jmx_exporter metrics, I met these error as below:

	at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
  at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
  at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
  ... 19 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
  javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308)
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
  at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344)
  at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
  at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134)
  ... 24 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
  at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719)
  at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518)
  at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
  at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
  at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
  at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1295)
  at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
  at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
  at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:128)
  at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:230)
  ... 28 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
  at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
  at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
  at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
  at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
  at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
  ... 36 more

configuration

jmx_exporter image version: 0.20.0
running mode: jmx_prometheus_httpserver
config.yaml

jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmxPort }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: true
username: admin
password: {{ default "" .Values.jmxPassword | quote }}

Doug Hoard · Answer 1 · Fri Apr 19 2024 22:29:53 GMT+0800 (China Standard Time)

When running the exporter in standalone mode with JMX being protected by SSL you have to...

Create a trust store to be used by the exporter.
Added the trusted certificate to the trust store )if you are not using a certificate signed by a public certificate authority)
Added the following properties to your exporter command line

-javax.net.ssl.trustStore=<file> -javax.net.ssl.trustStorePassword=<trustStore password>

whisper · Answer 2 · Mon Apr 22 2024 11:18:38 GMT+0800 (China Standard Time)

When running the exporter in standalone mode with JMX being protected by SSL you have to...

Create a trust store to be used by the exporter.

Added the trusted certificate to the trust store )if you are not using a certificate signed by a public certificate authority)

Added the following properties to your exporter command line
-javax.net.ssl.trustStore=<file> -javax.net.ssl.trustStorePassword=<trustStore password>

Thanks for your prompt reply. And do I need to add the keystore command line?

-Djavax.net.ssl.keyStore=/home/user/.keystore
-Djavax.net.ssl.keyStorePassword=changeit

Doug Hoard · Answer 3 · Mon Apr 22 2024 19:28:15 GMT+0800 (China Standard Time)

I don't believe the keyStore values are required.

whisper · Answer 4 · Tue Apr 23 2024 16:07:24 GMT+0800 (China Standard Time)

I don't believe the keyStore values are required.

Okay，I generated truststore.p12 through tomcatJMX.cer file, and specify trustStore in commandline, but still got the above memtioned error.

Create truststore
keytool -import -keystore ./truststore.p12 -storepass changeit-noprompt -trustcacerts -v -alias jmxssl -file ./tomcatJMX.cer
Running command:
java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit

Doug Hoard · Answer 5 · Tue Apr 23 2024 21:21:50 GMT+0800 (China Standard Time)

This command line is incorrect...

java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit

This is passing -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit as arguments to the exporter.

The correct command line should be...

java -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml

The use of the Java agent is strongly recommended. Some JVM metrics can't be captured when running the standalone exporter.

whisper · Answer 6 · Wed Apr 24 2024 09:21:37 GMT+0800 (China Standard Time)

This command line is incorrect...
java -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit
This is passing -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit as arguments to the exporter.

The correct command line should be...
java -Djavax.net.ssl.trustStore=./truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -jar jmx_prometheus_httpserver.jar 49103 /etc/jmx/jmx-server-prometheus.yaml
The use of the Java agent is strongly recommended. Some JVM metrics can't be captured when running the standalone exporter.

It seems that the agent mode cannot be used because I deployed the java application and jmx_exporter in a kubernetes environment.

Doug Hoard · Answer 7 · Wed Apr 24 2024 09:47:00 GMT+0800 (China Standard Time)

Deployment of the JMX Exporter Java agent works in Kubernetes.

whisper · Answer 8 · Wed Apr 24 2024 10:30:09 GMT+0800 (China Standard Time)

Hi @dhoard , I tried again based on the correct command you provided. Unfortunately, I still got the same error. Then, I checked the SSL related source code and found that after I deleted this line and tested it in local, I was able to pass the SSL authentication successful. This Is it a bug?

Doug Hoard · Answer 9 · Thu Apr 25 2024 01:08:27 GMT+0800 (China Standard Time)

@kingEneru I am reviewing #947, which appears to fail a new integration test I have created (not yet merged into main.)

EDIT: The code in main works correctly.

whisper · Answer 10 · Thu Apr 25 2024 09:19:30 GMT+0800 (China Standard Time)

@kingEneru I am reviewing #947, which appears to fail a new integration test I have created (not yet merged into main.)

EDIT: The code in main works correctly.

That's weird...
Does SSL authentication require that the jdk version of jmx exporter is the same as the jdk version of jmx server?

Doug Hoard · Answer 11 · Thu Apr 25 2024 12:08:59 GMT+0800 (China Standard Time)

That's weird... Does SSL authentication require that the jdk version of jmx exporter is the same as the jdk version of jmx server?

It does not. This is a configuration issue. I just updated PR #947 with the missing Java system property.

Doug Hoard · Answer 12 · Thu Apr 25 2024 13:44:02 GMT+0800 (China Standard Time)

@kingEneru When configuring RMI for SSL, the expectation is that the RMI registry is also configured for SSL. This requires the Java system property...

  -Dcom.sun.management.jmxremote.registry.ssl=true

... to be defined when launching your application.

I have merged an integration test that tests/validates RMI with SSL. @unitsvc also validated that adding the Java system property resolves the issue.

whisper · Answer 13 · Thu Apr 25 2024 15:15:09 GMT+0800 (China Standard Time)

@kingEneru When configuring RMI for SSL, the expectation is that the RMI registry is also configured for SSL. This requires the Java system property...
  -Dcom.sun.management.jmxremote.registry.ssl=true
... to be defined when launching your application.

I have merged an integration test that tests/validates RMI with SSL. @unitsvc also validated that adding the Java system property resolves the issue.

Okay, It's working successfully and normally now. Thanks a lot