Jenkins is unable to contact update center
targos opened this issue · comments
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
Full stack trace available on https://ci.nodejs.org/manage/pluginManager/advanced
BTW, the update site was https://updates.jenkins.io/experimental/update-center.json. I changed it to https://updates.jenkins.io/update-center.json
Note: ci-release is not affected.
Still a problem after #3637 and a full system package update + reboot.
There were OS updates, including one for the JRE, so I applied them all and rebooted.
The problem is still present.
@nodejs/build-infra any idea?
Interesting. Do you know if this message started appearing just after the update to 2.440?
(I'm going to be performing a similar update on another server tomorrow!)
It happened before the update to 2.440.1.
It looks like it's Java related somehow.
On ci.nodejs.org:
curl -sL https://updates.jenkins.io/update-center.jsonworks.
Using the example simple Java HTTPS Client from https://access.redhat.com/solutions/973783 (note I had to recompile the client with javac JavaHttpsClient.java as I couldn't directly run the class file from the archive):
java JavaHttpsClient https://updates.jenkins.io/update-center.json 1or (with debug info)
java -Djavax.net.debug=all JavaHttpsClient https://updates.jenkins.io/update-center.json 1fails.
On ci-release.nodejs.org both curl and java JavaHttpsClient succeed.
FWIW I've placed the JavaHttpsClient in /tmp/javahttpsclient on both machines.
ci.nodejs.org:
root@infra-digitalocean-ubuntu14-x64-1:~# lsb_release -d
Description: Ubuntu 22.04.4 LTS
root@infra-digitalocean-ubuntu14-x64-1:~# java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-122.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-122.04.1, mixed mode, sharing)
root@infra-digitalocean-ubuntu14-x64-1:~#vs ci-release.nodejs.org:
root@infra-ibm-ubuntu1804-x64-1:~# lsb_release -d
Description: Ubuntu 20.04.6 LTS
root@infra-ibm-ubuntu1804-x64-1:~# java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-120.04.1)
OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-120.04.1, mixed mode, sharing)
root@infra-ibm-ubuntu1804-x64-1:~#Given that curl worked on both machines, I don't think it should make a difference, but ci-release is on IBM Cloud whereas ci.nodejs.org is Digital Ocean.
Examining the debug output
ci.nodejs.org:
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.048 EST|TrustStoreManager.java:162|Inaccessible trust store: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.049 EST|TrustStoreManager.java:113|trustStore is: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
trustStore type is: pkcs12
trustStore provider is:
the last modified time is: Tue Jun 20 16:36:33 EDT 2023
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.049 EST|TrustStoreManager.java:334|Reload the trust store
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.127 EST|TrustStoreManager.java:342|Reload trust certs
javax.net.ssl|DEBUG|10|main|2024-03-08 13:09:40.130 EST|TrustStoreManager.java:347|Reloaded 173 trust certsci-release.nodejs.org:
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.022 UTC|TrustStoreManager.java:162|Inaccessible trust store: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.023 UTC|TrustStoreManager.java:113|trustStore is: /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
trustStore type is: pkcs12
trustStore provider is:
the last modified time is: Fri May 26 06:13:49 UTC 2023
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.023 UTC|TrustStoreManager.java:334|Reload the trust store
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.112 UTC|TrustStoreManager.java:342|Reload trust certs
javax.net.ssl|DEBUG|10|main|2024-03-08 18:08:59.116 UTC|TrustStoreManager.java:347|Reloaded 129 trust certsBoth try for an inaccessible (file isn't there /usr/lib/jvm/java-17-openjdk-amd64/lib/security/jssecacerts) before falling back to /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts. ci has loaded more certificates (173) than ci-release (129).
ci.nodejs.org:
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Jan 17 06:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 206799 Jun 20 2023 /etc/ssl/certs/java/cacerts
root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient#ci-release.nodejs.org:
root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts
lrwxrwxrwx 1 root root 27 Jan 17 11:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts
root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 150808 May 26 2023 /etc/ssl/certs/java/cacerts
root@infra-ibm-ubuntu1804-x64-1:/tmp/javahttpsclient#Yeah it looks like it's related to ubuntu system stuff. It seems to work ok with Temurin's 17.0.10 (and 21 - I've extracted both under your directory if you want to test).
Interesting. Do you know if this message started appearing just after the update to 2.440?
This issue was opened on 19 Feb. If I look at the certificates in my web browser for https://updates.jenkins.io/current/update-center.json
it looks like the current certificate was issued on 17 Feb -- my suspicion as to what has triggered this is https://letsencrypt.org/2023/07/10/cross-sign-expiration.html which had this happen earlier in the month that would change the default certificate chain:
On Thursday, Feb 8th, 2024, we stopped providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.
The JavaHttpsClient debug output shows both machines getting a two certificate chain.
ci.nodejs.org:root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts lrwxrwxrwx 1 root root 27 Jan 17 06:09 /usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts -> /etc/ssl/certs/java/cacerts root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient# ls -al /etc/ssl/certs/java/cacerts -rw-r--r-- 1 root root 206799 Jun 20 2023 /etc/ssl/certs/java/cacerts root@infra-digitalocean-ubuntu14-x64-1:/tmp/javahttpsclient#
The size of /etc/ssl/certs/java/cacerts on ci.nodejs.org differs from other Ubuntu 22.04 machines in our CI:
$ ssh test-digitalocean-ubuntu2204-x64-1 ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 156996 Nov 6 12:45 /etc/ssl/certs/java/cacerts
$ ssh test-digitalocean-ubuntu2204-x64-2 ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 156996 Nov 6 12:48 /etc/ssl/certs/java/cacerts
$ ssh ci ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 206799 Jun 20 2023 /etc/ssl/certs/java/cacerts
$I took a backup of /etc/ssl/certs/java/cacerts in /tmp/javahttpsclient/backup and ran:
(from https://askubuntu.com/a/1030380)
sudo dpkg --purge --force-depends ca-certificates-java
sudo apt-get install ca-certificates-javaThis has replaced /etc/ssl/certs/java/cacerts:
# ls -al /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 160820 Mar 11 13:53 /etc/ssl/certs/java/cacerts
#which still isn't the same size as the other Ubuntu 22.04 systems (🤷) but does allow java JavaHttpsClient https://updates.jenkins.io/update-center.json 1 to connect successfully (🎉).
I've restarted Jenkins on ci.nodejs.org so that Java picks up the refreshed certificates and the warning has gone away on https://ci.nodejs.org/manage/pluginManager/advanced:
Amazing work, thank you!