s_client seems not using client certificate when connecting via proxy
ftasnetamot opened this issue · comments
Tried with current debian based system OpenSSL 3.0.11 19 Sep 2023 (Library: OpenSSL 3.0.11 19 Sep 2023)
Connecting to FINAL-DESTINATION over proxy works perfect, as long, as the connection requires no client certificate.
openssl s_client -proxy PROXY:3128 -connect FINAL-DESTINATION:3128 results in no problem at all
However, using client certificate, the server claims a missing cert
openssl s_client -proxy PROXY:3128 -cert public.cert -key private.pem -connect FINAL-DESTINATION:3128
CONNECTED(00000003)
s_client: HTTP CONNECT failed, reason= 403 Forbidden
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 1448 bytes and written 73 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
As my test setup has also direct access to the FINAL-DESTINATION, I try the very same connection again.
This time just ommiting the proxy statement. Now everything works as expected.
openssl s_client -cert public.cert -key private.pem -connect FINAL-DESTINATION:3128
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E6
verify return:1
depth=0 CN = FINAL-DESTINATION
verify return:1
---
Certificate chain
0 s:CN = FINAL-DESTINATION
i:C = US, O = Let's Encrypt, CN = E6
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: Jun 18 09:23:10 2024 GMT; NotAfter: Sep 16 09:23:09 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = E6
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
....
WtUmIOeVwI0UAPEUAUBpG3yVn3ChgIA9d1/0mRabE3oxp6lowXD8Xw==
-----END CERTIFICATE-----
subject=CN = FINAL-DESTINATION
issuer=C = US, O = Let's Encrypt, CN = E6
---
Acceptable client certificate CA names
C = DE, ST = Some-State, O = Internet Widgits Pty Ltd, OU = none, CN = none, emailAddress = foo@bar.foo
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2604 bytes and written 1067 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 9868FB5922FF136A956B6903AAAAAAAAAAAAAAAAAAAA
Session-ID-ctx:
Resumption PSK: E463BFB6A90E98F0A3097D3CD322CA8BDE44341E5AAAAAAAAAAAAAAAAAAAAAAA
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 18000 (seconds)
TLS session ticket:
0000 - 57 45 39...
0010 - 8c 09 db...
Start Time: 1718960480
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 059667699806F7882560AAAAAAAAAAAAAAAAAAAA
Session-ID-ctx:
Resumption PSK: 6B08DA686877F291E22ADAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 18000 (seconds)
TLS session ticket:
0000 - 0a 67 ef d5 .....
0010 - e0 1b 7e .....
Start Time: 1718960480
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
So I see:
- -proxy statement works
- -client certificate work
- the combination of both does not work
Have I overseen something, or is this a bug?
what do the logs on your server say when the connection fails?
If I had to guess, I would hypothesize that your proxy is rewriting the source address of the packets being sent to your server, such that the server is not able to validate the supplied certificate against the ip address it sees at the source. I.e.:
- Using proxy alone is fine, because no client cert is supplied, and the server doesn't care about the origin of the connection
- Using a client cert directly is fine, because the cert name matches the ip address being used when the server validates the client cert
- Using both together is broken because the proxied packets not longer match the cert
@nhorman: This guess is wrong, as
- The client cert is not tied to an ip address, as it is an self signed cert with a user name as Common Name
- I tested the standalone connection also from the proxy server
- In one test client and proxy resided on the same host, also the same outgoing ip
- The server logs the same, as it reports back: That no certificate was sent
Can you post the certificate and server logs of the successful and failing connections here?
What is the proxy server?
There are many proxy servers that do not form a end-to-end connection (and in which case client authentication does not work).
Finally I got things working, but have not found (yet) the core culprit.
I have not longer looked at the proxy, as from my understanding the proxy could not be the culprit.
My current understanding is, that the proxy already did its job, by forwarding the connection afer receiving CONNECT. The proxy hands the tcp-stream over to openssl s_client and may had already a look in the next packets and therefore allowed TLS. So from my understanding, as soon, as the TCP handshake was succesful, the proxy should not know of client certs or whatever is going on.
Now I did some more steps, I describe here, as it may help others as well, fiddling with similar problems.
Separating the proxy from s_client
In the first step, I took out the proxy function from s_client, by establishing an socat connection, doing the proxy-job behind a local port.
socat TCP4-LISTEN:8080,fork PROXY:PROXY-IP:FINAL-DESTINATION:3128,proxyport=3128
and than started s_client with verfication-debugging, as this does not break connection, when verification fails
openssl s_client -verify 1 -cert ./public.cert -key ./private.pem -connect 127.0.0.1:8080
With this method, I could be sure, that openssl will use the cert params for shure for the TLS connections, as it is not aware of the proxy.
Creating another certificate set
As that also failed, I started to create a test-certificate for posting it to @nhorman, as I could not post the productive cert. To be conservative, I generated that now rsa based, and not ed25519, as the productive one. As this cert also should be tested, i configured the destination also to accept this cert and tested it with all setups.
And I got an connection!
mitmproxy
Ok, now I had to dig into the proxy stuff a little bit deeper. I used the mitmproxy, as this was already installed on my test-hostmitmproxy --ignore-hosts FINAL-DESTINATION-NAME, FINAL-DESTINATION-IP
This works for the ignored hosts as an pure forwarding and not intercepting proxy. And with this setup, both keys worked!
squid research
So squid on the pfsense firewall seems to be the culprit. I double checked the basic configuration, which contained not a single extra. Finally I followed the advice, I found in some forums posts, removing the packet, wiping all configuration remainders and reinstall. After I did this, I configured it just the same way, accepting connections from three interfaces, allow all users in the related networks, don't intercept TLS, don't cache and no other restrictions. And (for the further research unfortunately) now squid is working like the mitmproxy did it before, even with ed25519 certs.
So thanks to @nhorman and @t-j-h for forcing me that way. However, I am still a little bit clueless, what this squid instance has done! I can confirm: This is not a bug from openssl, thanks for the help!