Docker run - strange fatal error (TLS related?)
clmcavaney opened this issue · comments
Attempting to start a Docker container today from the latest version on the docker hub and I get the following error in the logs:
[07:33:23 INF] Loaded with configuration from: appsettings.json, /app/config/appsettings.Production.json
[07:33:23 FTL] error:2006D002:BIO routines:BIO_new_file:system lib
Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
at HomeAutio.Mqtt.GoogleHome.Startup.ConfigureServices(IServiceCollection services) in /app/HomeAutio.Mqtt.GoogleHome/Startup.cs:line 236
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at HomeAutio.Mqtt.GoogleHome.Program.Main(String[] args) in /app/HomeAutio.Mqtt.GoogleHome/Program.cs:line 45
Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
at HomeAutio.Mqtt.GoogleHome.Startup.ConfigureServices(IServiceCollection services) in /app/HomeAutio.Mqtt.GoogleHome/Startup.cs:line 236
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass15_0.<BuildStartupServicesFilterPipeline>g__RunPipeline|0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.StartupLoader.ConfigureServicesDelegateBuilder`1.<>c__DisplayClass14_0.<ConfigureServices>g__ConfigureServicesWithContainerConfiguration|0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.WebHost.Initialize()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at HomeAutio.Mqtt.GoogleHome.Program.Main(String[] args) in /app/HomeAutio.Mqtt.GoogleHome/Program.cs:line 45
Is there something wrong with the latest build?
Working here. Its failing in loading the signing certificates, but the error is cryptic as its coming out of the OpenSSL interop stuff directly from .NET Core.
Based on some stuff I see searching around it seems .NET is handing you the direct error from OpenSSL here, which while unclear, seems might be related to permissions errors on the cert?
Thanks for that - strange, as Docker is running as root and the files are owned by root. Nonetheless, changed permissions to allow world readable.
Now I just need to convert the TLS certificates to PKCS12 format for the Windows libraries used by HomeAutio.Mqtt.GoogleHome.
I thought I had it with a command like:
openssl pkcs12 -export -in <client-name>.crt -inkey <client-name>.key -out <client-name>.pfx -name GoogleHome -CAfile /etc/mosquitto/certs/ca.crt -caname My-CA -chain
But the container isn't able to connect (but I can with the PEM certificates from the command line with mosquitto_sub).
Will do further investigation on this.
Getting this in the logs:
[02:23:38 ERR] Error while connecting with server.
[02:23:38 INF] Disconnected.
[02:23:38 WRN] MQTT Connection failed, retrying...
[02:23:38 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:23:43 ERR] Error while connecting with server.
[02:23:43 INF] Disconnected.
[02:23:43 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:23:43 WRN] MQTT Connection failed, retrying...
[02:23:48 ERR] Error while connecting with server.
[02:23:48 INF] Disconnected.
[02:23:48 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:23:48 WRN] MQTT Connection failed, retrying...
[02:23:53 ERR] Error while connecting with server.
...
Actually - now I read that error message again, it looks like a connect denied not a TLS issue.
I'm investigating that further.
Wasn't able to determine why the connection is closed unexpectedly.
It would be good to have more detailed debugging information - is there a way to turn on more debugs?
Not around the MQTT connection. That's the logs from the underlying library being forwarded out.
It would be useful to confirm if the certificate I converted to PKCS12 from PEM format is actually working of if there is some other issue. Are there tools (like openssl) that I can use to determine if the .NET library is having an issue with the certificate?
I found some doco (https://techdrawer.wordpress.com/2014/10/07/testing-ssl-with-your-certificate-using-s_client/) and converted the PKCS12 versions back to PEM and then tested with the mosquitto_sub client and it worked.
openssl pkcs12 -in googlehome-apexnet.pfx -out client.pem -nodes
openssl pkcs12 -in googlehome-apexnet.pfx -out client.key -nodes -nocerts
openssl pkcs12 -in ca.pfx -out ca.pem -nodes
Then tested with s_client:
openssl s_client -connect mqtt.<mydomain>:8883 -cert client.pem -key client.key -CAfile ca.pem
Then tested with mosquitto_sub:
mosquitto_sub -v -h mqtt.<my domain> -p 8883 --cafile ca.pem --cert client.pem --key client.key -t 'homie/wemos-d1-mini-02/#'
homie/wemos-d1-mini-02/$state ready
homie/wemos-d1-mini-02/$homie 3.0.1
...
So, I think that confirms the PKCS12 files are valid for HomeAutio.Mqtt.GoogleHome to use.
In your other issue, your docker commands had --user=1000, which means your docker container ISN'T running as root as you said above, which would be the source of your original permissions issue. That doesn't need to be world readable, it needs to be owned by whoever user 1000 is on your system.
Note also that this is not for cert AUTHENTICATION its for MQTT over TLS.... looking at your mosquitto_sub command it looks like you might be trying to authenticate using this cert (I dont see username / pass specified there)? The underlying library I don't think even supports that yet. So the certs you can supply here are just for TLS connection cert chain validation.
I actually run with this, since I use a self signed cert and I didn't actually need to verify the chain for a local-to-local connection (acceptable risk).
"brokerTlsSettings": {
"allowUntrustedCertificates": true
}
By chance have you tried restarting mosquitto too? I had an issue not long ago where it refused to connect and I couldn't figure out why until I restarted mosquitto and everything went back to working... at the time I chalked it up to the certificate being updated (mine attempt to update automatically every night).
I restarted mosquitto but it still did not connect. I will triple check the certificates and try again.
And you aren't trying to authenticate using this certificate?
Correct, I changed the config to this:
"mqtt": {
"brokerIp": "mqtt.<my domain>",
"brokerPort": 8884,
"brokerUsername": "googlehome",
"brokerPassword": "",
"brokerUseTls": true,
"brokerTlsSettings": {
"allowUntrustedCertificates": true,
"ignoreCertificateChainErrors": true,
"ignoreCertificateRevocationErrors": false,
"protocol": "1.2",
"certificates": [
{
"file": "config/ca.pfx",
"passPhrase": "0bff79726c"
}
]
}
},
So again, this is NOT for certificate AUTHENTICATION, it is for MQTT over TLS. There is no way to use certificates for authentication with this app. You have to use username / pass.
Righto - will add password and try.
I need to have the CA certificate in the certificates section for the TLS though don't I?
Yes that should work, but I haven't done that myself. I just set the allowUntrustedCertificates setting, and the cert isn't actually validated, just used for trusted encryption.
Can I ask, what configuration do you have on your MQTT broker? I am using mosquitto and have:
cafile <my CA cert>
keyfile <my server key file>
certfile <my server cert file - signed by my CA cert>
allow_anonymous false
password_file
It is that CA cert file I have on the host with the docker container with HomeAutio.Mqtt.GoogleHome.
I have converted that CA cert file from PEM to PKCS12 format.
I believe that is correct.
But when the container starts, it is unable to connect to the remote MQTT broken.
I change config to non TLS and it connects fine, it is something to do with the TLS that is failing.
(the settings are as above (#92 (comment)) with a brokerPassword now)
"mqtt": {
"brokerIp": "1.1.1.1",
"brokerPort": 8883,
"brokerUsername": "username",
"brokerPassword": "password",
"brokerUseTls": true,
"brokerTlsSettings": {
"allowUntrustedCertificates": true
}
Note I use a letsencrypt cert for my MQTT over TLS encryption cert though too which isn't exactly a self-signed cert. The additional "ignore*" properties you have give you options to ignore various issues with certificate chain validations.
I am unclear on if the certificates are used only for cert authentication, or if they are also used as a source for CA certs for connection cert validation... Im actually asking the MQTT lib authors on that as conflating the two seems like it would be ill advised, and if they AREN'T conflated then having you set auth cert's in the area you set connection TLS settings seems confusing.
Note what I said here, the underlying library claims to support auth cert, using these exposed settings actually, so it SHOULD be possible as is... you might need to exclude username / password for it to try TLS auth instead though (unclear).
How do you convert your LetsEncrypt cert from PEM format to PKCS12 format for HomeAutio.Mqtt.GoogleHome to use?
I don't. As you can see in my config above, it has no awareness of the certificate, it's just set to accept whatever cert that Mosquitto is using for TLS (i.e. its ignoring all cert chain issues and it just accepting any cert it happens to be using).
Mosquitto accepts PEM just fine for its TLS setup, so no conversion necessary, I just copy the cert over for it to use.
Ahh yeah - sorry I missed that point. Of course, it should just ignore validation of the certificate.
So, that raises the question why I am getting this over and over (I am using the configuration you suggested above to allowUntrustedCertificates):
[06:32:08 ERR] Error while connecting with server.
[06:32:08 INF] Disconnected.
[06:32:08 INF] MQTT Connection closed unexpectedly, reconnecting...
[06:32:08 WRN] MQTT Connection failed, retrying...
[06:32:14 ERR] Error while connecting with server.
[06:32:14 INF] Disconnected.
[06:32:14 INF] MQTT Connection closed unexpectedly, reconnecting...
[06:32:14 WRN] MQTT Connection failed, retrying...
[06:32:19 ERR] Error while connecting with server.
[06:32:19 INF] Disconnected.
...
I did read that there is some subject name issues with .NET implementation of the TLS - so maybe it could be that - but there isn't enough debug information to isolate that as a potential issue.
Im not sure. In the config you posted above you're trying to connect to port 8884 instead of 8883 though... that on purpose?
Yes on purpose, I setup another listener in Mosquitto that didn't require/use client certificates for the username.
So, just TLS with basic username/password authentication.
The 8883 is configured for TLS with client certificates
Is there a way to turn on more detailed debugs for the TLS MQTT connection?
Not from this side. The underlying MQTT library does have an additional trace logging capability that might have more, but thats not something you have access to here without compiling a custom version.
Does Mosquitto's log not have anything in it? Thats probably the easiest place to look and see if you can get additional logging...
https://github.com/chkr1011/MQTTnet/wiki/Trace
You can TRY this if you want to compile and run an instance locally. I register the logging handler for their regular logs, which is what you see in my logs (I just forward them), but trace is a whole nother story. You could take the above and put it in the constructor of the MqttService class and forward the trace messages to the _log.Debug of that class to do the same thing... I can consider adding a configurable option for that, but it isn't going to be soon, because I'd have to update the base packages, and would want to upgrade all of my other apps to take advantage too, so you need this soon for debugging, that might be the easiest... you'd only really have to copy your config file over since you'd really only need the MQTT configuration.
G'day,
I put on some logging with Mosquitto and this is what shows up when the Docker container starts:
1603351892: New connection from **REDACTED IP ADDRESS** on port 8885.
1603351892: Socket error on client <unknown>, disconnecting.
1603351897: New connection from **REDACTED IP ADDRESS** on port 8885.
1603351897: Socket error on client <unknown>, disconnecting.
I stopped it after that.
The configuration I have in Mosquitto is on this port for TLS only with authentication via username/password.
I don't have a .Net compilation setup in place, haven't mucked around with .Net before - so I wouldn't be able to compile a different version up.
Ok, couple of things we can try:
What does openssl think of your cert?
openssl s_client -connect x.x.x.x:8885
Does one of these (insecure) work but the other doesn't? Note if you are using a self-signed cert you might need to sub in cafile for the certchain instead. I am using a LetsEncrypt cert so it validates on the default system CA certs, however I am ALSO using a cert tied to my DynDNS domain as SUBJECT, and connecting via direct IP, so I don't actually validate on the first command (domain / subject mismatch), but pass fine on the second one since it skips the cert checks.
mosquitto_sub -h x.x.x.x -p 8885 -v -d -u admin -P 'password' -t "#" --capath /etc/ssl/certs
mosquitto_sub -h x.x.x.x -p 8885 -v -d -u admin -P 'password' -t "#" --capath /etc/ssl/certs --insecure
OpenSSL s_client connects fine with only this error code (as expected):
Verify return code: 19 (self signed certificate in certificate chain)
Both the mosquitto_sub commands work - I eluded to that back here (albeit that was a client certificate example) #92 (comment)
Also, I am connecting to the DNS entry which matches the CN in the certificate subject.
This is why I am at a loss, as it should be a simple exercise to tell the .Net TLS code to connect and ignore the certificate authority verification. This what you suggested and I have put in place with the application configuration with this line in the "brokerTlsSettings" settings.
"allowUntrustedCertificates": true,
Your original mosquitto_sub commands ALSO were using pre-shared keys for auth though. The above commands SHOULD be equivalent to how this app is going to connect either WITHOUT allowUntrustedCertificates or WITH allowUntrustedCertificates. If both of those commands actually work for you, then I don't have any other answers at the moment, because I can't reproduce your issue at all. The only other ideas I have are tied to mosquitto configuration issues like usage of use_identity_as_username or use_subject_as_username, etc. I run a very stock mosquitto that really only sets up the TLS listener, the password authentication, and some logging options. Basically, the only options I set for mosquitto are the following. Are you using any additional mosquitto config options?
allow_anonymous false
password_file /mosquitto/config/passwd
listener 1883
listener 8883
certfile CERT
cafile CERT
keyfile CERT
log_dest file /mosquitto/log/mosquitto.log
log_dest stdout
persistence true
persistence_location /mosquitto/data/
Just checked my mosquitto conf, except for the ports almost (paths are the only diff) exactly the same.
If I open 1883 to the world and test the remote container it connects fine.
Something in the TLS is causing issue.
I will see if I can try a LetsEncrypt certificate and see if that makes any difference (shouldn't but it is the only real different in setups). i.e. I have a self signed cert, and you have a LetsEncrypt cert.
Note I was playing with something else yesterday, and you might try setting the log level to Verbose and see if the MQTT library spits out anything else useful to you... I noticed when I turned it on to test something else that it looked like there might have been some MQTT related additional log items with that...
Well well well, I put a Let's Encrypt certificate in place on the Mosquitto server configuration for the port 8885 listener.
Restarted the Docker container and voilà it connected to the MQTT server without error.
Changed the Mosquitto back to self signed certificate configuration for that port.
Restarted the Docker container and it didn't connect.
It makes me wonder if the allowUntrustedCertificates is being ignore or something.
I will turn on the verbose settings and see if it shows anything else, but at least there is a potential solution.
Turned on Verbose logs and interesting findings.
I started the container, which connected successfully as the MQTT server had the Let's Encrypt certificates in place.
I then change to self signed and restarted the MQTT server (whilst the Docker container was still running), then saw this in the logs:
[02:03:10 VRB] Stopped sending keep alive packets.
[02:03:10 VRB] Disconnecting [Timeout=00:00:10]
[02:03:10 VRB] Disconnected from adapter.
[02:03:10 INF] Disconnected.
[02:03:10 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:03:10 VRB] Stopped receiving packets.
[02:03:10 VRB] Trying to connect with server 'mqtt.**REDACTED**:8885' (Timeout=00:00:10).
[02:03:11 ERR] Error while connecting with server.
[02:03:11 VRB] Disconnecting [Timeout=00:00:10]
[02:03:11 VRB] Disconnected from adapter.
[02:03:11 INF] Disconnected.
[02:03:11 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:03:11 WRN] MQTT Connection failed, retrying...
[02:03:11 VRB] Stopped publishing messages.
[02:03:16 VRB] Trying to connect with server 'mqtt.**REDACTED**:8885' (Timeout=00:00:10).
[02:03:16 ERR] Error while connecting with server.
[02:03:16 VRB] Disconnecting [Timeout=00:00:10]
[02:03:16 VRB] Disconnected from adapter.
[02:03:16 INF] Disconnected.
...
changed back to Let's Encrypt certificates, then this:
[02:03:51 INF] MQTT Connection closed unexpectedly, reconnecting...
[02:03:51 WRN] MQTT Connection failed, retrying...
[02:03:56 VRB] Trying to connect with server 'mqtt.**REDACTED**:8885' (Timeout=00:00:10).
[02:03:57 VRB] Connection with server established.
[02:03:57 VRB] Start receiving packets.
[02:03:57 VRB] TX (100 bytes) >>> Connect: [ClientId=85839bbf-**REDACTED**] [Username=googlehome] [Password=****] [KeepAlivePeriod=15] [CleanSession=True]
[02:03:57 VRB] RX (4 bytes) <<< ConnAck: [ReturnCode=ConnectionAccepted] [ReasonCode=] [IsSessionPresent=False]
[02:03:57 VRB] Authenticated MQTT connection with server established.
[02:03:57 INF] Connected.
[02:03:57 INF] MQTT Connection established
[02:03:57 INF] Publishing subscriptions at reconnect
[02:03:57 VRB] TX (67 bytes) >>> Subscribe: [PacketIdentifier=1] [TopicFilters=google/home/commands/+/set@AtLeastOnce,homie/wemos-d1-mini-02/light/on@AtLeastOnce]
[02:03:57 VRB] Start sending keep alive packets.
[02:03:57 VRB] RX (6 bytes) <<< SubAck: [PacketIdentifier=1] [ReturnCodes=SuccessMaximumQoS1,SuccessMaximumQoS1] [ReasonCode=]
[02:03:57 VRB] TX (28 bytes) >>> Publish: [Topic=google/home/connected] [Payload.Length=1] [QoSLevel=AtLeastOnce] [Dup=False] [Retain=True] [PacketIdentifier=2]
[02:03:57 VRB] RX (38 bytes) <<< Publish: [Topic=homie/wemos-d1-mini-02/light/on] [Payload.Length=1] [QoSLevel=AtLeastOnce] [Dup=False] [Retain=True] [PacketIdentifier=1]
[02:03:57 INF] MQTT message received for topic homie/wemos-d1-mini-02/light/on: 0
So, nothing too much extra logged about the TLS handshake, but shows clearly how one connects but the other doesn't.
I traced it back all the way to the base library... I mean it certainly looks like its handing it back right at least in my code, and I verified its reading it from the config alright. Unless Im missing something, it could be an issue with the Mqtt library I use. I wonder if anyone else has gotten it working with a self-signed cert, I know I went right to a letsencrypt one.
You aren't by chance using the WebSockets connections are you?
G'day Michael,
No, I did play around a long time ago trying to get a WebSockets listener working, but it didn't work.
I was also wondering if there is a TLS version issue - but the testing today proves that can't be the case, as the Mosquitto configuration was default for that so whatever the .NET code negotiates would be the version used. The only difference was the certificates themselves.
Regards.
Christopher
Hm well, I'm stumped, but Ill keep this open until I have time to try some self-signed certs and hunt down where validation is failing.
Closing as unable to reproduce (and age).