Pre-KitKat (4.4) Android-versions gets shut out from application (java.security.cert.CertPathValidatorException: Trust anchor for certification path not found)

[pargu]

After installing plugin it is not possible to run our application from devices running versions lower than Android 4.4.

Environment:

Jelastic 4.9.5.3 (Elastx)
CentOS 7.2
Nginx 1.10.1

Server solution: Parse Server 2.2.24

Test-devices:

Google Nexus 6 (7.1)
Samsung Galaxy Tab A (6.0.1)
Sony Xperia Z5 Compact (5.1.1)
Motorola Moto X 2013 (4.4.4)
Samsung G313HN (4.4.2)
Galaxy Nexus (4.2.2)

Testing

1. Run tests on ssllabs on our domain and it states support down to 4.0... be available throughTLS 1.0 and these certificates seen in images below
Links:
http://ibb.co/hrWU8v
http://ibb.co/fDkZaa
https://www.ssllabs.com/ssltest/

2. Ran this command in terminal.
openssl s_client -connect domain-name:443

Got this result:

Certificate chain
0 s:/CN=domain-name
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
1 s:/CN=domain-name
i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
2 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Number 2 is available in all devices in Settings -> Security -> Trusted credentials. (Even on Galaxy Nexus, which is not working)

3. Checking Client side for Parse Android SDK fault
This is how Parse does HTTP requests throughout app:
if (hasOkHttpOnClasspath()) { httpClientLibraryName = OKHTTP_NAME; httpClient = new ParseOkHttpClient(socketOperationTimeout, sslSessionCache); } else if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) { httpClientLibraryName = URLCONNECTION_NAME; httpClient = new ParseURLConnectionHttpClient(socketOperationTimeout, sslSessionCache); } else { httpClientLibraryName = APACHE_HTTPCLIENT_NAME; httpClient = new ParseApacheHttpClient(socketOperationTimeout, sslSessionCache); }

Our concerns were the ParseApacheHttpClient, as it states if not OKHttp3 available and less than KitKat (4.4) it should hit that. This library is deprecated since API 20 and we had concerns this would be the issue.

After adding SDK locally and debugging the queries all versions hits hasOkHttp check. Even Galaxy Nexus 4.2.2. So this is not the issue

The Exception that gets thrown on < 4.4 is:
java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
According to Android documentation this can happen if:

1. The CA that issued the server certificate was unknown
2. The server certificate wasn't signed by a CA, but was self signed
3. The server configuration is missing an intermediate CA

1 should be the issue as the certificate is not self-signed and the server has an intermediate CA installed (see result in 2)

Links:
https://developer.android.com/training/articles/security-ssl.html#CommonProblems

4. Checking CA Certificates
So after this we started looking at Let's Encrypt certificates Chain of Trust and found out it's crossed-signed using IdenTrust (contrary to what 2, openssl command says pointing it to Digital Signature Trust Co./CN=DST Root CA X3) which should work on all Android versions.

Checking trusted CA Certificates on devices was the next step. We found that IdenTrust was available on these Devices:
Google Nexus 6
Samsung Galaxy Tab A

Not available on these:
Sony Xperia Z5 Compact
Samsung G313HN
Motorola Moto X
Galaxy Nexus (4.2.2)

So this could be ruled out.. Kinda.

Links:
https://letsencrypt.org/certificates/
https://www.identrust.com/support/download_root_cert.html

5. What we've tried

First try at solution:
Adding process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0" to index.js

We found out some have had issue with Parse Cloud Code when enabling HTTPS and found some issues on Parse Server github linking this command as a solution. We had not been experienced this issue with the Cloud code but gave it a try. It did nothing.

Second try at solution: (Temp fix)
Adding Custom CA Certificate to client:

1. Downloaded .pem file (x3) available at Lets Encrypt's certificates site.
2. run command in terminal to extract .crt file from .pem
3. Added file to ROOT on Galaxy Nexus through USB
4. Open Settings on device -> Security - Install from Storage
5. Device prompt Lets Encrypt certificate from ROOT and pressed OK
6. Go in to Trusted Certificates -> Choose User tab -> Lets Encrypt cert is showing.
7. Open application.
8. Log in.
9. Success.

So this is a temporary solution but not a good one. We can send this file to our customers, that gets affected by this, or see if they can update to 4.4. But we need this to be working automatically for all Android versions, as it states it should, running SSLLabs test.

Any thoughts? Is this related to the plugin somehow? As running Lets encrypt by adding it to device works, and it says it uses another CA certificate on the server. We've had some customers hitting this issue, and all devices are below 4.4 KitKat.

[ihorman]

Dear Pargu,

Thank you for reporting the issue. Did you try to use your own external domain or you were using elastx shared domain? The thing is that in with shared domain the test certificate is used so it couldn't be validated and trusted.

[pargu]

Hi ihorman,
We are using an external domain, with Public IP turned on. It's running on a VPS through CentOS

[ihorman]

Dear Pargu,

Could you please ping me in Skype (ihorman)? I am sure that problem is somewhere in a trust chain, I need your stand/environment, or at least link to it to check/debug? Thank you.

[ihorman]

fixed!
Thank you for reporting the issue ;)