DEFAULT_CERT_DIR not being used

[Chris Oliver]

Hey all! I ran into what seems like a bug with Ruby verifying SSL certs using the certs DIR.

Steps to reproduce

1. Create a self-signed certificate. In my case, I setup elasticsearch on Ubuntu server 22.04.
2. Copy the self-signed certificate to another Ubuntu 22.04 server.
3. Add the self-signed certificate to /usr/local/share/ca-certificates/elasticsearch.crt and run update-ca-certificates which trusts the cert
4. curl to the elasticsearch server works with https and verifies the certificate
5. Ruby fails

Context

/home/deploy/.asdf/installs/ruby/3.1.2/bin/ruby (3.1.2-p20)
OpenSSL 3.0.2 15 Mar 2022: /usr/lib/ssl

irb> OpenSSL::X509::DEFAULT_CERT_FILE
=> "/usr/lib/ssl/cert.pem"
irb> OpenSSL::X509::DEFAULT_CERT_DIR
=> "/usr/lib/ssl/certs"

On Ubuntu, /usr/lib/ssl/cert.pem does not exist by default, but /usr/lib/ssl/certs directory does. I confirmed the self-signed cert is hashed and symlinked in this directory after running update-ca-certificates.

$ ls -la /etc/ssl/certs | grep elasticsearch
lrwxrwxrwx 1 root root     21 Jul 25 22:34 0fac13e2.0 -> elasticsearch.crt
lrwxrwxrwx 1 root root     21 Jul 25 22:34 86a8db8f.0 -> elasticsearch.crt
-rw-r----- 1 root root   1914 Jul 25 21:50 elasticsearch.crt
lrwxrwxrwx 1 root root     54 Jul 25 22:29 elasticsearch.pem -> /usr/local/share/ca-certificates/elasticsearch.crt

I expect the SSL certificate to be verified using the DEFAULT_CERT_DIR. Instead, I get the following error when making the request:

require 'net/http'
Net::HTTP.get(URI("https://192.168.64.23:9200"))
OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 peeraddr=192.168.64.23:9200 state=error: certificate verify failed

Instead, it seems like the DEFAULT_CERT_DIR is ignored by OpenSSL::X509::Store#verify.

The GitLab team appears to have run into the same issue as documented here: https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/784

Something tells me there's something is "off" in store.verify because it is not consuming OpenSSL::X509::DEFAULT_CERT_DIR appropriately.

A random hunch: Is it possible this happens because Ubuntu's /usr/lib/ssl/certs is a symlink?

$ ls -la /usr/lib/ssl/
total 12
drwxr-xr-x  3 root root 4096 Jul 25 22:51 .
drwxr-xr-x 84 root root 4096 Jul 24 21:02 ..
lrwxrwxrwx  1 root root   14 Mar 16  2022 certs -> /etc/ssl/certs
drwxr-xr-x  2 root root 4096 Jul 18 21:30 misc
lrwxrwxrwx  1 root root   20 May 24 12:12 openssl.cnf -> /etc/ssl/openssl.cnf
lrwxrwxrwx  1 root root   16 Mar 16  2022 private -> /etc/ssl/private

However, testing this assumption it doesn't seem to work with SSL_CERT_DIR pointed to the directory (not symlink) either:

$ SSL_CERT_DIR=/etc/ssl/certs ruby -rnet/http -e 'Net::HTTP.get(URI("https://192.168.64.23:9200"))'
/home/deploy/.asdf/installs/ruby/3.1.2/lib/ruby/3.1.0/net/protocol.rb:46:in `connect_nonblock': SSL_connect SYSCALL returned=5 errno=0 peeraddr=192.168.64.23:9200 state=error: certificate verify failed (OpenSSL::SSL::SSLError)

Curious if this is a bug or if I'm just doing something wrong.

[Chris Oliver]

Using strace, I found a Permission denied error as it tried to read the cert.

strace -ff ruby -r"net/http" -e 'puts Net::HTTP.get(URI("https://HOST:PORT"))'

openat(AT_FDCWD, "/usr/lib/ssl/certs/0fac13e2.0", O_RDONLY) = -1 EACCES (Permission denied)

⚠️ When adding certs, make sure to chmod 644 certpath!

Here's how I'm adding certificates for anyone that stumbles across this:

echo "certificate" > /usr/local/share/ca-certificates/example.crt
chmod 644 /usr/local/share/ca-certificates/example.crt
update-ca-certificates