intermittent SSLHandshakeException: unable to find valid certification path to requested target retrieving IAM token
gridhawk opened this issue · comments
Getting issues loading files from Cloud Object Storage in the Java Spring application.
Our Spring service is running as a container in IBM Kubernetes in VPC.
We followed documentation from https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-java#java-examples to connect.
We access the bucket through app - Java - followed documentation on how to connect.
https://cloud.ibm.com/docs/vpc?topic=vpc-connecting-vpc-cos#how-to-connect-to-ibm-cloud-object-storage-cos-from-a-vpc
So the connection is ok, and we can list buckets, but when the app tries to get some resources in the bucket we get an error
We use in the app - https://s3.eu-de.cloud-object-storage.appdomain.cloud
we are using that way of connecting, our endpoint that we use is - s3.direct.eu-de.cloud-object-storage.appdomain.cloud
We are authenticating the same way from your docs - using api key: AWSCredentials credentials = new BasicIBMOAuthCredentials(apiKey, serviceInstanceId);
The problem is not related to Kafka, we are running Kafka as a container in Kubernetes. So we are not using Event Streams as IBM Service.
Our Java application just can't access the bucket after 60mins.
Just for your info, we also have NodeJS container, that uses IBM cos SDK for uploading files, and that works without issues, only Java application breaks on fetching files from bucket after 60mins from deploy.
It's still the same. We have looked at IBM documentation and followed everything, and can't find the problem.
The problem occurs sometimes, so 3 times app usually works, and then a couple of times we are getting following exception:
Caused by: com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
2022-08-29 09:53:31
2022-08-29 07:53:31,088 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] c.o.b.a.c.ExceptionHandlerAOP [ExceptionHandlerAOP.java:28] Unhandled exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
Our concern is, because it sometimes work and sometimes does not, what could be the cause of such behaviour?
So, last night at 22:44:41CEST was successful access.
This morning at 10:30:19 CEST we got the error
2022-08-29 10:30:19
Caused by: com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
recreate scenario:
We have managed to find a case how it breaks.
When new container is deployed it works fine for some time, then it logs:
2022-08-29 13:32:35,572 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-29 15:32:35
2022-08-29 13:32:35,531 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-29 15:32:05
2022-08-29 13:32:05,531 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-29 15:32:05
2022-08-29 13:32:05,491 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
after that it can't fetch anything from bucket
The problem is that after some time it stops working, and getting SSLHandshake error. We have noticed that it can't fetch Refresh token properly perhaps. Because when container is deployed and after some time (30-60mins) it stops working with SSLHandshake error, before that, it works perfectly.
here are some more logs, after 45mins this is what happened and logged:
Problem seems to be in refresh token, after this, getting items in bucket will not work until another container restart.
2022-08-30 15:39:49
2022-08-30 13:39:49,706 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-30 15:39:49
2022-08-30 13:39:49,663 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-30 15:39:19
2022-08-30 13:39:19,662 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-30 15:39:19
2022-08-30 13:39:19,604 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-30 15:38:49
2022-08-30 13:38:49,603 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-30 15:38:49
2022-08-30 13:38:49,564 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-30 15:38:19
2022-08-30 13:38:19,563 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-30 15:38:19
2022-08-30 13:38:19,522 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-30 15:37:49
2022-08-30 13:37:49,522 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-30 15:37:49
2022-08-30 13:37:49,484 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
We now see the SSLHandshake error with MongoDB as well. We use Database for MongoDB, and it was working fine, but now it's failing with same error.
Our COS team has reviewed this case and have stated that these requests are not reaching the endpont,
this issue is present after some time after the container is deployed (it works for 60-90 minutes then starts to error), and it is resolved by restarting the container,
We have another container in a cluster in VPC. That container has NodeJS with IBM SDK - ibm-cos-sdk and it works fine. Once deployed it never crashes.
The problem is with Java Spring app and ibm-cos-java-sdk. App failes on Refresh token for some reason. We use eclipse-temurin:18-jdk-alpine image to run spring boot app.
Another thing - we have noted that the problem is on the line:
ObjectListing listing = cosClient.listObjects(bucketName, radarId);
so after 60mins, app tries to get files in the bucket and we get:
Error reading COS bucketReceived javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
So how can it connect when the container is deployed but can't fetch RefreshToken?
Because, if app is IDLE we get:
[2022-08-31 23:59:20,158] INFO [pool-1-thread-1] com.ibm.cloud.objectstorage.oauth.DefaultTokenManager OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-09-01 01:59:20
[2022-08-31 23:59:20,091] INFO [pool-1-thread-1] com.ibm.cloud.objectstorage.oauth.DefaultTokenManager OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
We have resolved our issue with AWS SDK instead of using IBM COS SDK for Java.
We configured AWS SDK to use config from IBM COS buckets and it is working fine now.
So the problem must be with IBM COS SDK for Java somewhere.
cstask0044646 The issue is not reaching to endpoint, but after deploying the container it works for sometime and then they get ssl timeout errors.
@gridhawk
An internal ticket has been created to track the issue and we will be working on it.
Requesting customer contact to gather the following:
- which ibm-cos-sdk version is used.
- part of the java code that is causing the error.
here is the info:
com.ibm.cos
ibm-cos-java-sdk
2.12.0
- The scenario is:
when container is being deployed, it connects to bucket without issues with code:
AWSCredentials credentials = new BasicIBMOAuthCredentials(apiKey, serviceInstanceId);
ClientConfiguration clientConfiguration = new ClientConfiguration()
.withRequestTimeout(5000)
.withTcpKeepAlive(true);
return AmazonS3ClientBuilder
.standard()
.withCredentials(new AWSStaticCredentialsProvider(credentials))
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpointUrl, location))
.withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfiguration)
.build();
and then we list bucket:
cosClient.listBuckets();
all works fine.
and then after ~ 60mins we get: SSLHandshake when we try:
ObjectListing listing = cosClient.listObjects(bucketName, radarId);
error is:
com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
2022-08-29 09:53:31
2022-08-29 07:53:31,088 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] c.o.b.a.c.ExceptionHandlerAOP [ExceptionHandlerAOP.java:28] Unhandled exception: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target) (Service: IAM; Status Code: 0; Error Code: null; Request ID: null; S3 Extended Request ID: null; Proxy: null)
also we get:
2022-08-29 13:32:35,572 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-29 15:32:35
2022-08-29 13:32:35,531 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
2022-08-29 15:32:05
2022-08-29 13:32:05,531 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Exception retrieving IAM token. Returned status code 0. Error Message: Received javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target retrieving IAM token (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target). Status Message: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
2022-08-29 15:32:05
2022-08-29 13:32:05,491 INFO [pool-1-thread-1] c.i.c.o.o.DefaultTokenManager [CommonsLog.java:63] OAuthTokenManager.TokenRefreshTask: Attempting to retrieve refresh token
*Update: we tried various ways to recreate the customer issue using the exact code provided, but we are not facing any such issue or error.
Currently we suspect the problem [ after ~ 60mins we get: SSLHandshake ] is moreover an issue between SSL and server and less related to SDK.
Setting case as resolved with workaround in place and no further requests from involved development team to require customer input.
Also closing github #55 as an internal ticket is in place for tracking this issue.
In case of any new findings we will reach out to you then.
Let me give you some more info:
Our container is running in k8s with VPC. So we have 2 containers:
- Spring - using ibm-cos-sdk causing this issue
- NodeJS - using ibm-cos-sdk for nodejs without any issues
So maybe problem is somewhere with JVM and certificates to look into.