SSLHandshake issue after connection timeout

Question

SSLHandshake issue after connection timeout

sbuda opened this issue 8 years ago · comments

Hi, I am using your package to access data from Quandl

Last weekend, due to some unknown issue, the connection timed out whilst running the following code:

public void retrieveColumns(LocalDate startDate, LocalDate endDate, String mySymbol) {
    QuandlSession session = QuandlSession.create("My Key");
    TabularResult tabularResult = session.getDataSet(
                    DataSetRequest.Builder
                        .of(mySymbol)
                        .withFrequency(Frequency.DAILY)
                        .withStartDate(getQuandleDate(startDate))
                        .withEndDate(getQuandleDate(endDate))
                        .withSortOrder(SortOrder.DESCENDING)
                        .build()
                );
}

I assume this was due to a blip in the internet, or Quandl experiencing downtime. What was interesting was what happened after. From this point on, every time our application tried to make a request, we got the following error:

javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching www.quandl.com found.
    at org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:229) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:655) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:652) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:652) ~[jersey-client-2.8.jar:na]
    at com.jimmoores.quandl.util.DefaultRESTDataProvider.getTabularResponse(DefaultRESTDataProvider.java:89) ~[quandl-1.3.0.jar:na]
    at com.jimmoores.quandl.QuandlSession.getDataSet(QuandlSession.java:159) ~[quandl-1.3.0.jar:na]
    ...
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching www.quandl.com found.
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_71]
    ...
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching www.quandl.com found.
    at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:204) ~[na:1.8.0_71]
    …

Once we restarted the application, things worked fine again. What I want to know is, why did this issue occur, and if there is anything I can do resolve this issue programmatically if it happens again. I emailed Quandl support, but they basically said "Everything looks fine to us, contact us if it happens again."

Obviously, I'd like to be a little more proactive about it. I'm sorry I can't give you a reproducible sample for this, the events that caused it seem to be out of my hands. Any help would be appreciated.

Jim Moores · Answer 1 · Thu Jun 02 2016 23:07:00 GMT+0800 (China Standard Time)

I'll be honest, I'm not sure. I suspect this is Java caching a bad certificate returned by Quandls DNS setup until you restarted. What version of Java are you using? Older versions can have out of date certificate lists. I've had SSL configuration problems with Quandl before revolving around their certificate being registered against www.quandl.com but not quandl.com (or something like that, I forget the details). Hopefully it's transient, but I'm guessing you'll have to catch the error and somehow force a flush of the certificate cache at the application level. I'm happy to include such logic if I had more information.

sbuda · Answer 2 · Thu Jun 02 2016 23:50:14 GMT+0800 (China Standard Time)

Thanks for looking into this! Is there any information in particular that I can provide you to help out?

sbuda · Answer 3 · Fri Jun 03 2016 02:55:11 GMT+0800 (China Standard Time)

Oh, and to answer your Question, our server says it is running:

openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)

So it should be close to the newest version as far as I know.

sbuda · Answer 4 · Thu Jun 09 2016 00:23:25 GMT+0800 (China Standard Time)

For what it's worth, we got a similar issue again last night, however the errors in the stacktrace were different. As with last time, first we got a connection timeout, indicating that Quandl was unavailable. Afterwards, the error this time was:

javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused
    at org.glassfish.jersey.client.HttpUrlConnector.apply(HttpUrlConnector.java:229) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:224) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:655) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:652) ~[jersey-client-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424) ~[jersey-common-2.8.jar:na]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:652) ~[jersey-client-2.8.jar:na]
    at com.jimmoores.quandl.util.DefaultRESTDataProvider.getTabularResponse(DefaultRESTDataProvider.java:89) ~[quandl-1.3.0.jar:na]
    at com.jimmoores.quandl.QuandlSession.getDataSet(QuandlSession.java:159) ~[quandl-1.3.0.jar:na]
  ...
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_71]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_71]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_71]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_71]
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_71]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_71]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.8.0_71]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.8.0_71]
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[na:1.8.0_71]
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[na:1.8.0_71]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[na:1.8.0_71]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) ~[na:1.8.0_71]
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:989) ~[na:1.8.0_71]
    at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:987) ~[na:1.8.0_71]
    ...

Again, restarting the server resolved the issue. From looking around on the web, a lot of people suggest this happens when looking at the wrong port. Either way, it seems like Quandl changes something on their end, and then java doesn't work. Unfortunately, it seems like there may be multiple points of failure, if at first I had issues with the certificate cache, and later with the actual ip/port.

Do you have any suggestions on how I can flush, clean, restart, or whatever needs done here without restarting the server itself?

Jim Moores · Answer 5 · Thu Jun 09 2016 00:47:12 GMT+0800 (China Standard Time)

Hi,

First thing you could try is customising the retry policy, which you can do when creating a session by passing a SessionOptions to the create method. It supports fixed number of retries or a repeating retry forever. The default is 5 retries - each increasing in time up to one minute. This seemed when I wrote it to handle most of the unreliability, but things could have changed. It's possible the retry mechanism isn't catching the connection refused exception, which should be fixable, but I'm not sure if that's happening or not. Lastly I (or you) could modify the retry loop on each call to move the Client and Target objects inside each loop to create fresh ones each retry rather than reuse. That might help with the certificate issue as well.

I'm on vacation at the moment, which limits what I can do, but I'll be back in London next week though.

Jim

sbuda · Answer 6 · Thu Jun 09 2016 00:49:12 GMT+0800 (China Standard Time)

Hi Jim,

Thanks again for your input. I'll try around some things and see what happens. Enjoy your vacation and I'll let you be :)