TLS handshake failed., System.IO.IOException

Question

TLS handshake failed., System.IO.IOException

AnOdyssey opened this issue 4 months ago · comments

AnOdyssey commented 4 months ago

Expected Behavior

Module sending data to iothub.

Current Behavior

Getting TLS handshake errors between the edgeHub and a custom C# module when trying to connect to the mqqt.

Output of `iotedge check`

Click here


Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
31 check(s) succeeded.
4 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

Host OS: Ubuntu 22.04
Architecture: amd64
Container OS: Docker

Runtime Versions

aziot-edged: iotedge 1.4.27
Edge Agent: 1.4
Edge Hub: 1.4
Docker/Moby: 24.0.7-1

Logs

logs from the module:

[08:39:57 WRN] Connection changed: Status: Disconnected_Retrying Reason: Communication_Error
[08:39:57 WRN] Connection changed: Status: Disconnected Reason: Retry_Expired

aziot-edged logs

Partially omitted due to github content length. This part contains the only visible error.

jan 10 09:33:38 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:38Z [INFO] - Watchdog checking Edge runtime status
jan 10 09:33:38 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:38Z [INFO] - Edge runtime is running
jan 10 09:33:39 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:39Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:33:39 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:39Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:33:44 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:44Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:33:44 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:44Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:33:49 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:49Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:33:49 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:49Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:33:54 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:54Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:33:54 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:54Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:33:59 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:59Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:33:59 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:33:59Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:04 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:04Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:04 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:04Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:09 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:09Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:09 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:09Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:14 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:14Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:14 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:14Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:19 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:19Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:19 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:19Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:21 dev-Virtual-Machine aziot-certd[31454]: 2024-01-10T08:34:21Z [INFO] - <-- GET /certificates/aziot-edged-trust-bundle?api-version=2020-09-01 {"host": "certd.sock"}
jan 10 09:34:21 dev-Virtual-Machine aziot-certd[31454]: 2024-01-10T08:34:21Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [ERR!] - error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:../crypto/encode_decode/decoder_lib.c:101:No supported data to decode. Input type: PEM
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [ERR!] - !!! internal error
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [ERR!] - !!! caused by: could not load key pair
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [ERR!] - !!! caused by: could not load key pair: AZIOT_KEYS_RC_ERR_EXTERNAL
jan 10 09:34:21 dev-Virtual-Machine aziot-keyd[31397]: 2024-01-10T08:34:21Z [INFO] - --> 500 {"content-type": "application/json"}
jan 10 09:34:24 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:24Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:24 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:24Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:24 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:24Z [INFO] - <-- GET /modules/?api-version=2018-06-28 {"host": "unix:///var/run/iotedge/mgmt.sock", "connection": "Close"}
jan 10 09:34:24 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:24Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:29 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:29Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:29 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:29Z [INFO] - --> 200 {"content-type": "application/json"}
jan 10 09:34:34 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:34Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 10 09:34:34 dev-Virtual-Machine aziot-edged[31393]: 2024-01-10T08:34:34Z [INFO] - --> 200 {"content-type": "application/json"}

edge-agent logs


<6> 2024-01-10 08:28:16.216 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:28:16.216 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:28:16.216 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:28:16.511 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:28:16.512 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:28:16.512 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bModBusModule\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:28:16.519 +00:00 [INF] - Initiating streaming logs for ModBusModule
<6> 2024-01-10 08:28:16.561 +00:00 [INF] - Received 1861 bytes of logs for ModBusModule
<6> 2024-01-10 08:28:16.561 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:28:31.700 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:28:31.701 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:28:31.701 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:28:31.788 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:28:31.789 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:28:31.789 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeHub\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:28:31.795 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2024-01-10 08:28:31.796 +00:00 [INF] - Received 437 bytes of logs for edgeHub
<6> 2024-01-10 08:28:31.797 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:30:18.988 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:30:18.989 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:30:18.989 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:30:19.065 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:30:19.066 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:30:19.066 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bModBusModule\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:30:19.073 +00:00 [INF] - Initiating streaming logs for ModBusModule
<6> 2024-01-10 08:30:19.142 +00:00 [INF] - Received 2003 bytes of logs for ModBusModule
<6> 2024-01-10 08:30:19.143 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:35:26.892 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:35:26.892 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:35:26.893 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:35:26.984 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:35:26.984 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:35:26.984 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeHub\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:35:26.989 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2024-01-10 08:35:26.990 +00:00 [INF] - Received 586 bytes of logs for edgeHub
<6> 2024-01-10 08:35:26.991 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:35:37.165 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:35:37.166 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:35:37.166 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:35:37.237 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:35:37.237 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:35:37.237 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bModBusModule\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:35:37.244 +00:00 [INF] - Initiating streaming logs for ModBusModule
<6> 2024-01-10 08:35:37.319 +00:00 [INF] - Received 1991 bytes of logs for ModBusModule
<6> 2024-01-10 08:35:37.319 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:36:06.089 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:36:06.089 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:36:06.089 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:36:06.182 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:36:06.182 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:36:06.183 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeAgent\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:36:09.136 +00:00 [INF] - Initiating streaming logs for edgeAgent
<6> 2024-01-10 08:36:09.157 +00:00 [INF] - Received 1800 bytes of logs for edgeAgent
<6> 2024-01-10 08:36:09.158 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:36:44.732 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:36:44.732 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:36:44.732 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:36:44.827 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:36:44.828 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:36:44.828 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeHub\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:36:44.832 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2024-01-10 08:36:44.834 +00:00 [INF] - Received 602 bytes of logs for edgeHub
<6> 2024-01-10 08:36:44.834 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:40:11.234 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:40:11.234 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:40:11.234 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:40:11.310 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:40:11.311 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:40:11.311 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeHub\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:40:11.317 +00:00 [INF] - Initiating streaming logs for edgeHub
<6> 2024-01-10 08:40:11.318 +00:00 [INF] - Received 647 bytes of logs for edgeHub
<6> 2024-01-10 08:40:11.319 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:41:01.960 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:41:01.960 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:41:01.961 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:41:02.046 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:41:02.046 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:41:02.046 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bModBusModule\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}
<6> 2024-01-10 08:41:02.053 +00:00 [INF] - Initiating streaming logs for ModBusModule
<6> 2024-01-10 08:41:02.128 +00:00 [INF] - Received 2126 bytes of logs for ModBusModule
<6> 2024-01-10 08:41:02.128 +00:00 [INF] - Successfully handled request GetModuleLogs
<6> 2024-01-10 08:43:07.568 +00:00 [INF] - Received direct method call - ping
<6> 2024-01-10 08:43:07.568 +00:00 [INF] - Received request ping with payload
<6> 2024-01-10 08:43:07.568 +00:00 [INF] - Successfully handled request ping
<6> 2024-01-10 08:43:07.648 +00:00 [INF] - Received direct method call - GetModuleLogs
<6> 2024-01-10 08:43:07.648 +00:00 [INF] - Received request GetModuleLogs with payload
<6> 2024-01-10 08:43:07.649 +00:00 [INF] - Processing request to get logs for {"schemaVersion":"1.0","items":{"id":"\\bedgeAgent\\b","filter":{"tail":1500,"since":"15m","until":null,"loglevel":null,"regex":""}},"encoding":1,"contentType":1}

edge-hub logs

<6> 2024-01-10 08:25:17.959 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2024-01-10 08:26:00.333 +00:00 [INF] - Obtained edge hub config patch update from module twin
<6> 2024-01-10 08:26:00.377 +00:00 [INF] - Updating edge hub configuration
<6> 2024-01-10 08:26:00.398 +00:00 [INF] - Set the following 1 route(s) in edge hub
<6> 2024-01-10 08:26:00.398 +00:00 [INF] - ModBusModuleToIoTHub: FROM /messages/modules/ModBusModule/outputs/* INTO $upstream
<6> 2024-01-10 08:26:00.398 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2024-01-10 08:26:00.398 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2024-01-10 08:26:00.551 +00:00 [INF] - Updated reported properties for Arno-dev/$edgeHub
<4> 2024-01-10 08:27:13.667 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 79206572"
<6> 2024-01-10 08:30:17.959 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2024-01-10 08:31:14.708 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 3a488cc2"
<4> 2024-01-10 08:32:15.057 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 7f59e19c"
<4> 2024-01-10 08:34:17.347 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 73e8128d"
<6> 2024-01-10 08:35:17.329 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint iothub
<6> 2024-01-10 08:35:17.334 +00:00 [INF] - Cleaned up 0 messages from queue for endpoint iothub and 0 messages from message store.
<6> 2024-01-10 08:35:17.958 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2024-01-10 08:35:21.015 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 37368612"
<4> 2024-01-10 08:36:27.235 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 20939f9e"
<4> 2024-01-10 08:37:37.240 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 1e2411c6"
<4> 2024-01-10 08:38:47.745 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 6671ae08"
<4> 2024-01-10 08:39:57.755 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 79c2f250"

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

Gaurav Kulkarni · Answer 1 · Thu Jan 11 2024 05:52:19 GMT+0800 (China Standard Time)

@AnOdyssey - please try deleting and regenerating your keys.

If that fails, check to see if this is what you're running into: openssl/openssl#16696

AnOdyssey · Answer 2 · Thu Jan 11 2024 15:24:26 GMT+0800 (China Standard Time)

Hey @gauravIoTEdge, thanks for the quick response.

What keys are you referring to? We followed these instructions from Microsoft to setup a single local dev device(Raspberry pi, local vm, windows docker). There is no mention of key generation. If you mean the keys it generates its selve during setup this is the 3th time installing it on a different device resulting in the same behavior.

This is the openssl result when connecting to the internal mqtt on port 8883

OpenSSL result

CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = dev-Virtual-Machine
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = dev-Virtual-Machine
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = dev-Virtual-Machine
verify return:1
---
Certificate chain
 0 s:CN = dev-Virtual-Machine
   i:CN = aziot-edge CA dev-dev
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 11 07:17:11 2024 GMT; NotAfter: Feb 10 07:17:11 2024 GMT
-----BEGIN CERTIFICATE-----
MIIDPTCCAiWgAwIBAgIBADANBgkqhkiG9w0BAQsFADAhMR8wHQYDVQQDDBZhemlv
dC1lZGdlIENBIEFybm8tZGV2MB4XDTI0MDExMTA3MTcxMVoXDTI0MDIxMDA3MTcx
MVowHzEdMBsGA1UEAwwUYXJuby1WaXJ0dWFsLU1hY2hpbmUwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCiSj/a7cmg5IhrCgacJLOWhUJk1SwYgymamqJ3
Hq6M88ScOUMa7pkOC5SOGYkkd2+YNWoMB1iSUjc+adA0glX0202O3N8jVadUpvAC
4H99KdgXLYbLy8JJMcnLsaOvecaUgmRQa8qojbotCCFRMGJYNvFpvTqA7Xm0PRp0
IzqXwj4n98BvS5cfD66VekDm3AXWegMBdt0r6oi5aCkQBrg6Fhz7SnXx0rYnvB0h
UA5CblE234DJXi+dlGV6neMIvFPlhk5JwldcdxW6KwLvgsxBijprqBE32S8KnLRD
ik6kCcWEtzpQpEPBmUhsvVScpwoJgnaQhbwkwTjtjh2aXe4dAgMBAAGjgYEwfzAT
BgNVHSUEDDAKBggrBgEFBQcDATAoBgNVHREEITAfghRhcm5vLVZpcnR1YWwtTWFj
aGluZYIHZWRnZUh1YjAdBgNVHQ4EFgQU7UW6cngJUrN5Yn2N7AwicXRmbMEwHwYD
VR0jBBgwFoAUTQnHwD956g4HY01dpYGssO1eAXAwDQYJKoZIhvcNAQELBQADggEB
AIerOUERB8sf5YBQ2RWbuzp0ShXjwQ/S+bRKLncTa5Ka/DM7dnkMMtmqdK66yPPy
TgxfksDQFUhTsdEws0Pz9ySXZs7iAPaXlm4cQuHFbzG+TVW43znJvTlOTx/jfbY2
Vu4g9fcxSFZDIqmtoA/pFw1JRHUphosrphkiWhvDSksSVwRdY2Pq6FtywJWrUYxG
D+88CPc8vfZPjSgt7CJsDRsr+QJtPht1rqLel6csTrmV24NGyfid/4a++21ruesk
zu6XkzqshE2oyWOqBn1SKMBe13ycSP9EfJgdOtT/IOT62J92svslH7xT3eVlQe3C
b+BKGjrvAP6Z8PzR2/xvNFw=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = dev-Virtual-Machine
issuer=CN = aziot-edge CA dev-dev
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1515 bytes and written 398 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: B980356B0808C1D49172AB9A93E3C0B95AE8C2874403BFAECB303DEB961FAB5C
    Session-ID-ctx:
    Master-Key: 8A32BAD38CFF280E9D92B102BD3EF0E19B08D1080B559FADF2D2CFC06A5B5959939431203277E5F5F033277DE9D1B10D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 21 57 e4 a0 8f fa 56 4f-65 46 80 7f 6a 16 ae 70   !W....VOeF..j..p
    0010 - 45 44 93 04 34 cb 05 6f-64 42 95 69 5c e4 6b fd   ED..4..odB.i\.k.
    0020 - 84 c8 62 e8 e0 43 57 8d-da 6e c1 c3 6c 26 84 8b   ..b..CW..n..l&..
    0030 - ac ee 4d 63 21 e3 ce d7-e1 17 e3 0c 4d e4 a7 bc   ..Mc!.......M...
    0040 - 70 9a 29 96 a4 7e dc 9f-b5 ce 17 fe 52 7f dd e0   p.)..~......R...
    0050 - b4 6a 2c b0 b7 2c c3 fb-e3 0e 07 28 fc 95 ea ff   .j,..,.....(....
    0060 - f6 b2 54 bf 36 c6 71 dd-ea a5 6f 7f 9a 95 85 73   ..T.6.q...o....s
    0070 - ae 66 fa fa 96 2f c9 0b-d5 13 35 6c b9 46 c2 06   .f.../....5l.F..
    0080 - 0b 7b c4 35 a3 56 69 f2-4f f9 18 83 11 3c 01 cb   .{.5.Vi.O....<..
    0090 - 0b 79 41 66 c1 43 18 b2-e5 cd e9 fa 9f 27 ab da   .yAf.C.......'..

    Start Time: 1704957636
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes

Gaurav Kulkarni · Answer 3 · Fri Jan 12 2024 03:22:09 GMT+0800 (China Standard Time)

Actually, I meant your key-pair @AnOdyssey. The X.509 certs.

Have you been using the same-key pair by any chance? It's worth checking to see if a new key pair will work.

For context, the aziot-edged logs show this:

And maybe it's an Edge issue, but it looks an awful lot like that openssl issue I linked.

With this one, I'm not sure - it's worth just trying with a new key-pair, if possible.

AnOdyssey · Answer 4 · Fri Jan 12 2024 03:46:28 GMT+0800 (China Standard Time)

For this development deployment we have not generated any keys ourselves since we are using symmetric keys to deploy the IoT Edge. If I understand correctly following this Microsoft article. The edge will use the connection string to retrieve one of the keys to authenticate itself.

That being said. There is no issue between IoT Hub and the IoT Edge, we are perfectly getting the status of the edge modules and are able to retrieve the logs.

Do we need to generate a cert ourselves for the internal mqtt hub that the edge hosts? If so is there any documentation for this since it seem to not be mentioned anywhere in the symmetric key deployment article.

Gaurav Kulkarni · Answer 5 · Fri Jan 12 2024 05:26:58 GMT+0800 (China Standard Time)

Ah, I see.

Thanks for the added details @AnOdyssey

I understand you've tried to install this on 3 devices, but I'd appreciate you trying the following on one of those 3 devices rather than a clean install. I want to see if this makes a difference. (In a perfect world, you should not have to do this - but please try once and let us know?)

It looks like the key used by edgeHub is corrupted or missing. I suggest recreating the edgeHub module and regenerating keys.

Stop iotedge: sudo iotedge system stop
Delete edgeHub: sudo docker container rm edgeHub
If you have persisted edgeHub's storage directory, delete everything in that directory
Go to /var/lib/aziot/keyd/keys and delete the files starting with aziotidentitydmasterid-, handlevalidationkey-, and iotedgemasterencryptionid-.
Restart iotedge: sudo iotedge sytem restart

AnOdyssey · Answer 6 · Fri Jan 12 2024 15:31:47 GMT+0800 (China Standard Time)

I've done the commands as suggested. However we are still seeing the same issue happening. I'll include the logs.
In the iotedge system logs I'm seeing openssl errors appear again.

iotedge system logs


jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Received signal; shutting down
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Watchdog stopped
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping all modules...
jan 12 07:49:33 dev-Virtual-Machine systemd[1]: Stopping Azure IoT Edge daemon...
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping module ModBusModule...
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping module edgeHub...
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping module edgeAgent...
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping listener for module ModBusModule
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping listener for module edgeHub
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping listener for module edgeAgent
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Workload API stopped
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Workload API stopped
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Workload API stopped
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - <-- POST /modules/ModBusModule/stop?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping module ModBusModule...
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - Stopping listener for module ModBusModule
jan 12 07:49:33 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:33Z [INFO] - <-- POST /modules/edgeHub/stop?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
jan 12 07:49:43 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:43Z [INFO] - --> 204 {}
jan 12 07:49:43 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:43Z [INFO] - Stopping module edgeHub...
jan 12 07:49:43 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:43Z [INFO] - Stopping listener for module edgeHub
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - --> 204 {}
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - All modules stopped
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - watchdog finished
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Stopping management API...
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Stopping workload API...
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Management API stopped
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Workload Manager stopped
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Stopping listener for module ModBusModule
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Stopping listener for module edgeHub
jan 12 07:49:53 dev-Virtual-Machine aziot-edged[917]: 2024-01-12T06:49:53Z [INFO] - Stopping listener for module edgeAgent
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: aziot-edged.service: Deactivated successfully.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopped Azure IoT Edge daemon.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopping Azure IoT Identity Service...
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: aziot-identityd.service: Deactivated successfully.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopped Azure IoT Identity Service.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopping Azure IoT Keys Service...
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: aziot-keyd.service: Deactivated successfully.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopped Azure IoT Keys Service.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopping Azure IoT Certificates Service...
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: aziot-certd.service: Deactivated successfully.
jan 12 07:49:54 dev-Virtual-Machine systemd[1]: Stopped Azure IoT Certificates Service.
jan 12 07:53:26 dev-Virtual-Machine systemd[1]: Started Azure IoT Edge daemon.
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting Azure IoT Edge Daemon
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Version - 1.4.27
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Obtaining Edge device provisioning data...
jan 12 07:53:26 dev-Virtual-Machine systemd[1]: Started Azure IoT Identity Service.
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Starting service...
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Version - 1.4.7
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Loaded openssl'd Default provider
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Provisioning starting. Reason: Startup
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Updated device info for dev-dev.
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Provisioning complete.
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Identity reconciliation started. Reason: Startup
jan 12 07:53:26 dev-Virtual-Machine systemd[1]: Started Azure IoT Keys Service.
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - Starting service...
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - Version - 1.4.7
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - Loaded libaziot-keys with version 0x02010000
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - Starting server...
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "370"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Identity reconciliation complete.
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - Starting server...
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Device is dev-dev on IH-xxx-dev.azure-devices.net
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Initializing module runtime...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Using runtime network id azure-iot-edge
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Successfully initialized module runtime
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"}
jan 12 07:53:26 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine systemd[1]: Started Azure IoT Certificates Service.
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - Starting service...
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - Version - 1.4.7
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - Starting server...
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Using existing Edge CA certificate
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
jan 12 07:53:26 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:26Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-22T08:04:38+00:00.
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping all modules...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping module ModBusModule...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping module edgeHub...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping module edgeAgent...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - All modules stopped
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Detecting if device information has changed...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Device information has not changed
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting management API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting workload API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting new listener for module ModBusModule
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting workload API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting new listener for module edgeHub
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting workload API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting new listener for module edgeAgent
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting workload API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping listener for module ModBusModule
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping listener for module edgeHub
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Stopping listener for module edgeAgent
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Workload API stopped
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Workload API stopped
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Workload API stopped
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting watchdog with 60 second period...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting image garbage collection task...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Watchdog checking Edge runtime status
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Edge runtime status is failed, starting module now...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting module edgeAgent...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting new listener for module edgeAgent
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Starting workload API...
jan 12 07:53:26 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:26Z [INFO] - Started Edge runtime module edgeAgent
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - <-- GET /trust-bundle?api-version=2020-07-07 {"accept": "application/json", "host": "workload.sock:80", "connection": "close"}
jan 12 07:53:27 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:27Z [INFO] - <-- GET /certificates/aziot-edged-trust-bundle?api-version=2020-09-01 {"host": "certd.sock"}
jan 12 07:53:27 dev-Virtual-Machine aziot-certd[3271]: 2024-01-12T06:53:27Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - Querying system info...
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3392]: microsoft
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - Successfully queried system info
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - Querying system info...
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3400]: microsoft
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - Successfully queried system info
jan 12 07:53:27 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:27Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - Querying system info...
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3414]: microsoft
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - Successfully queried system info
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638403144337256133/decrypt?api-version=2022-08-03 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "1635"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "74"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "1967"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - OpenSSL error
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! internal error
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! caused by: could not decrypt
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! caused by: could not decrypt: AZIOT_KEYS_RC_ERR_EXTERNAL
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 500 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - --> 500 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638403144337256133/sign?api-version=2018-06-28 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "169"}
jan 12 07:53:28 dev-Virtual-Machine aziot-identityd[3259]: 2024-01-12T06:53:28Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "370"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638403144337256133/decrypt?api-version=2022-08-03 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "1635"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "74"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "1967"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - OpenSSL error
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! internal error
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! caused by: could not decrypt
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - !!! caused by: could not decrypt: AZIOT_KEYS_RC_ERR_EXTERNAL
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 500 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-edged[3257]: 2024-01-12T06:53:28Z [INFO] - --> 500 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [ERR!] - invalid parameter "id": not found
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - !!! a parameter has an invalid value
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 400 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /key?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "75"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - --> 200 {"content-type": "application/json"}
jan 12 07:53:28 dev-Virtual-Machine aziot-keyd[3261]: 2024-01-12T06:53:28Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "736"}

EdgeHub logs via azure portal


2024-01-12 06:54:19  Starting Edge Hub
2024-01-12 06:54:19  Starting Edge Hub
2024-01-12 06:54:19  Creating UID 13623 as edgehubuser
2024-01-12 06:54:19  Creating storage folder: /tmp/edgeHub
2024-01-12 06:54:19  Creating backup folder: /tmp/edgeHub_backup
2024-01-12 06:54:20.008 +00:00 Edge Hub Main()
<6> 2024-01-12 06:54:20.662 +00:00 [INF] - Installing certificates [CN=aziot-edge CA dev-dev:04/09/2024 08:04:38] to Root
<6> 2024-01-12 06:54:20.689 +00:00 [INF] - Installing certificates [CN=aziot-edge CA dev-dev:04/09/2024 08:04:38] to Root
<6> 2024-01-12 06:54:20.703 +00:00 [INF] - Enabling SSL protocols: Tls12
<6> 2024-01-12 06:54:20.769 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2024-01-12 06:54:20.992 +00:00 [INF] - Created persistent store at /tmp/edgeHub
<6> 2024-01-12 06:54:21.017 +00:00 [INF] - Initializing Edge Hub
<6> 2024-01-12 06:54:21.017 +00:00 [INF] - 
        â��â��â��â��â��â�� â��â��â��â��â��â��â��â��â��â��â��   â��â��â��â��â��â��â��â��â��â�� â��â��â��â��â��â��â��â��
       â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��   â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��
       â��â��â��â��â��â��â��â��  â��â��â��â��â�� â��â��â��   â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��
       â��â��â��â��â��â��â��â�� â��â��â��â��â��  â��â��â��   â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��
       â��â��â��  â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��  â��â��â��â��â��â��â��â��â��â��â��
       â��â��â��  â��â��â��â��â��â��â��â��â��â��â�� â��â��â��â��â��â��â�� â��â��â��  â��â��â��â��â��â��â��â��â��â��â��

 â��â��â�� â��â��â��â��â��â��â�� â��â��â��â��â��â��â��â��â��    â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��  â��â��â��â��â��â��â�� â��â��â��â��â��â��â��â��
 â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��    â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â�� â��â��â��â��â��â��â��â��
 â��â��â��â��â��â��   â��â��â��   â��â��â��       â��â��â��â��â��â��  â��â��â��  â��â��â��â��â��â��  â��â��â��â��â��â��â��â��â��â��
 â��â��â��â��â��â��   â��â��â��   â��â��â��       â��â��â��â��â��â��  â��â��â��  â��â��â��â��â��â��   â��â��â��â��â��â��â��â��â��
 â��â��â��â��â��â��â��â��â��â��â��â��   â��â��â��       â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��
 â��â��â�� â��â��â��â��â��â��â��    â��â��â��       â��â��â��â��â��â��â��â��â��â��â��â��â��â��â��  â��â��â��â��â��â��â�� â��â��â��â��â��â��â��â��

<6> 2024-01-12 06:54:21.017 +00:00 [INF] - Version - 1.4.29.85541525 (f049017a5072f85aec250e945ebb2ca52e4af3f4)
<6> 2024-01-12 06:54:21.018 +00:00 [INF] - OptimizeForPerformance=True
<6> 2024-01-12 06:54:21.018 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2024-01-12 06:54:21.018 +00:00 [INF] - Loaded server certificate with expiration date of "2024-02-11T06:54:20.0000000+00:00"
<6> 2024-01-12 06:54:21.022 +00:00 [INF] - Using Asp Net server for metrics
<6> 2024-01-12 06:54:21.048 +00:00 [INF] - Created new message store
<6> 2024-01-12 06:54:21.082 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2024-01-12 06:54:21.097 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2024-01-12 06:54:21.133 +00:00 [INF] - Created device scope identities cache
<6> 2024-01-12 06:54:21.137 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2024-01-12 06:54:21.149 +00:00 [INF] - Initialized storing twin manager
<6> 2024-01-12 06:54:21.163 +00:00 [INF] - Add node: dev-dev/$edgeHub
<6> 2024-01-12 06:54:21.164 +00:00 [INF] - Initializing configuration
<6> 2024-01-12 06:54:21.182 +00:00 [INF] - New device connection for device dev-dev/$edgeHub
<6> 2024-01-12 06:54:21.192 +00:00 [INF] - Client dev-dev/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2024-01-12 06:54:21.229 +00:00 [INF] - Attempting to connect to IoT Hub for client dev-dev/$edgeHub via AMQP...
<6> 2024-01-12 06:54:21.359 +00:00 [INF] - Processing pending subscriptions for dev-dev/$edgeHub
<6> 2024-01-12 06:54:21.735 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2024-01-12 06:54:21.779 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<4> 2024-01-12 06:54:21.788 +00:00 [WRN] - Empty edge hub configuration received. Ignoring...
<6> 2024-01-12 06:54:21.902 +00:00 [INF] - Scheduling server certificate renewal for "2024-02-11T06:51:50.0003859Z".
<6> 2024-01-12 06:54:22.116 +00:00 [INF] - Exiting disconnected state
<6> 2024-01-12 06:54:22.139 +00:00 [INF] - Received device connected callback
<6> 2024-01-12 06:54:22.146 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2024-01-12 06:54:22.147 +00:00 [INF] - Processing subscriptions for client dev-dev/$edgeHub on device connected to cloud.
<6> 2024-01-12 06:54:22.148 +00:00 [INF] - Skipping dev-dev/$edgeHub for subscription processing, as it is currently being processed.
<6> 2024-01-12 06:54:22.158 +00:00 [INF] - Entering connected state
<6> 2024-01-12 06:54:22.160 +00:00 [INF] - Cloud connection for dev-dev/$edgeHub is True
<6> 2024-01-12 06:54:22.165 +00:00 [INF] - Connection status for dev-dev/$edgeHub changed to ConnectionEstablished
<6> 2024-01-12 06:54:22.165 +00:00 [INF] - Client dev-dev/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2024-01-12 06:54:22.166 +00:00 [INF] - Skipping dev-dev/$edgeHub for subscription processing, as it is currently being processed.
<6> 2024-01-12 06:54:22.167 +00:00 [INF] - Created cloud proxy for client dev-dev/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-12 06:54:22.170 +00:00 [INF] - Initialized cloud proxy 237cb084-e6a0-4985-8ed5-da1491b07624 for dev-dev/$edgeHub
<6> 2024-01-12 06:54:22.173 +00:00 [INF] - Created cloud connection for client dev-dev/$edgeHub
<6> 2024-01-12 06:54:22.293 +00:00 [INF] - Add node: dev-dev
<6> 2024-01-12 06:54:22.344 +00:00 [INF] - Add node: dev-dev/$edgeAgent
<6> 2024-01-12 06:54:22.394 +00:00 [INF] - Updated node: dev-dev/$edgeHub
<6> 2024-01-12 06:54:22.413 +00:00 [INF] - Service identity for dev-dev/$edgeHub in device scope was updated.
<6> 2024-01-12 06:54:22.413 +00:00 [INF] - Add node: dev-dev/ModBusModule
<6> 2024-01-12 06:54:22.429 +00:00 [INF] - Add node: dev-dev/DefenderIotMicroAgent
<6> 2024-01-12 06:54:22.452 +00:00 [INF] - Processing pending subscriptions for dev-dev/$edgeHub
<6> 2024-01-12 06:54:22.454 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes.
<6> 2024-01-12 06:54:22.587 +00:00 [INF] - Updated reported properties for dev-dev/$edgeHub
<6> 2024-01-12 06:54:22.677 +00:00 [INF] - Created cloud endpoint iothub with max batch size 10 and fan-out factor of 10.
<6> 2024-01-12 06:54:22.730 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2024-01-12 06:54:22.821 +00:00 [INF] - Set the following 1 route(s) in edge hub
<6> 2024-01-12 06:54:22.821 +00:00 [INF] - ModBusModuleToIoTHub: FROM /messages/modules/ModBusModule/outputs/* INTO $upstream
<6> 2024-01-12 06:54:22.822 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2024-01-12 06:54:22.822 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2024-01-12 06:54:22.825 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2024-01-12 06:54:22.826 +00:00 [INF] - Initialized edge hub configuration
<6> 2024-01-12 06:54:22.826 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2024-01-12 06:54:22.828 +00:00 [INF] - Starting MQTT head
<6> 2024-01-12 06:54:22.848 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2024-01-12 06:54:22.884 +00:00 [INF] - Starting AMQP head
<6> 2024-01-12 06:54:22.923 +00:00 [INF] - Started AMQP head
<6> 2024-01-12 06:54:22.924 +00:00 [INF] - Starting HTTP head
<4> 2024-01-12 06:54:23.018 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
<6> 2024-01-12 06:54:23.024 +00:00 [INF] - Started HTTP head
<6> 2024-01-12 06:54:23.026 +00:00 [INF] - Started MQTT head
<6> 2024-01-12 06:54:26.316 +00:00 [INF] - Updated reported properties for dev-dev/$edgeHub
<4> 2024-01-12 06:55:23.112 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 34765899"
<4> 2024-01-12 06:56:23.100 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 1ad16d01"
<4> 2024-01-12 06:57:23.095 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 3f617bcc"
<4> 2024-01-12 06:58:23.102 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 0b9f6836"
<6> 2024-01-12 06:59:21.784 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<4> 2024-01-12 06:59:23.104 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 6481ed09"
<4> 2024-01-12 07:00:33.112 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 65f2d956"
<4> 2024-01-12 07:01:43.117 +00:00 [WRN] - "TLS handshake failed., System.IO.IOException: Channel is closed, 4f26c3e1"

Module warnings


[07:01:43 WRN] Connection changed: Status: Disconnected_Retrying Reason: Communication_Error
[07:01:43 WRN] Connection changed: Status: Disconnected Reason: Retry_Expired

I've looked at the keys after they were regenerated. aziotedgeca-. looks to be a full key. The other 3 keys you mentioned are only a couple of characters. Is this correct? Or are we expecting full private keys like aziotedgeca-?

Gaurav Kulkarni · Answer 7 · Sat Jan 13 2024 03:38:40 GMT+0800 (China Standard Time)

@AnOdyssey - Actually, this helped quite a bit. It's a different error.

I understand that what you want is for it to work and not have to do any of this. So, I'll explain the situation, as I see it:

I know you tried this on 3 devices. I'm only speaking to the one that you've reused and not the other 2 (because what you may be seeing is still a Connection changed: Status: Disconnected_Retrying Reason: Communication_Error in your logs across all devices, but underneath, in the iotedge system logs - the reasons for this error are different, and it matters).

When you first reported the issue, from our perspective, the problem was No supported data to decode. Input type: PEM in the iotedge logs. I linked it to this openssl issue: openssl/openssl#16696 -- and I still don't know if this was an iotedge problem or an openssl quirk.

When it became clearer that the problem is the edgeHub keys, I asked you to try and regenerate them.

Once you did that, we started seeing this in the iotedge system logs:

We typically see this error if you don't clean the storage directory and redeploy the module or the device, because redeploying creates a new identity and thus a new encryption key. Did you perhaps configure a storage directory?

If so, please delete it after step 2 in my previous comments. That should work.

@jlian - Even with the workaround, they will probably want to know why they saw the original error in the first place. That needs a deeper insight than I'm able to offer at this time. Could we please pull in someone from IIS?

AnOdyssey · Answer 8 · Sat Jan 13 2024 04:29:33 GMT+0800 (China Standard Time)

I would love for things to work first time, unfortunately this is not very often the case as developer :)

Personally I have not specified a storage director in our deployment or during iotedge configuration, iotedge check keeps on giving warnings as well that data might be lost due to there being no persistent storage defined. I'll include the deployment configuration. and iotedge check output.

deployment config


{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.1",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              "registry1": {
                "address": "redacted",
                "username": "redacted",
                "password": "redacted"
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.4",
              "createOptions": "{}"
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.4",
              "createOptions": "{\"HostConfig\":{\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
            }
          }
        },
        "modules": {
          "ModBusModule": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "redacted.azurecr.io/modbusmodule:0.0.1-arm64v8.debug",
              "createOptions": "{}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.2",
        "routes": {
          "ModBusModuleToIoTHub": "FROM /messages/modules/ModBusModule/outputs/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}

iotedge check


Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
31 check(s) succeeded.
4 check(s) raised warnings. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

AnOdyssey · Answer 9 · Tue Jan 16 2024 17:15:53 GMT+0800 (China Standard Time)

Found the issue!

It was related to the TLS version used by the internal generated certificates. Mainly for the mqtt as far as I know. I've posed this generated certificated before.

Generated certificate used by mqtt

CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = dev-Virtual-Machine
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = dev-Virtual-Machine
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = dev-Virtual-Machine
verify return:1
---
Certificate chain
 0 s:CN = dev-Virtual-Machine
   i:CN = aziot-edge CA dev-dev
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 11 07:17:11 2024 GMT; NotAfter: Feb 10 07:17:11 2024 GMT
-----BEGIN CERTIFICATE-----
MIIDPTCCAiWgAwIBAgIBADANBgkqhkiG9w0BAQsFADAhMR8wHQYDVQQDDBZhemlv
dC1lZGdlIENBIEFybm8tZGV2MB4XDTI0MDExMTA3MTcxMVoXDTI0MDIxMDA3MTcx
MVowHzEdMBsGA1UEAwwUYXJuby1WaXJ0dWFsLU1hY2hpbmUwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCiSj/a7cmg5IhrCgacJLOWhUJk1SwYgymamqJ3
Hq6M88ScOUMa7pkOC5SOGYkkd2+YNWoMB1iSUjc+adA0glX0202O3N8jVadUpvAC
4H99KdgXLYbLy8JJMcnLsaOvecaUgmRQa8qojbotCCFRMGJYNvFpvTqA7Xm0PRp0
IzqXwj4n98BvS5cfD66VekDm3AXWegMBdt0r6oi5aCkQBrg6Fhz7SnXx0rYnvB0h
UA5CblE234DJXi+dlGV6neMIvFPlhk5JwldcdxW6KwLvgsxBijprqBE32S8KnLRD
ik6kCcWEtzpQpEPBmUhsvVScpwoJgnaQhbwkwTjtjh2aXe4dAgMBAAGjgYEwfzAT
BgNVHSUEDDAKBggrBgEFBQcDATAoBgNVHREEITAfghRhcm5vLVZpcnR1YWwtTWFj
aGluZYIHZWRnZUh1YjAdBgNVHQ4EFgQU7UW6cngJUrN5Yn2N7AwicXRmbMEwHwYD
VR0jBBgwFoAUTQnHwD956g4HY01dpYGssO1eAXAwDQYJKoZIhvcNAQELBQADggEB
AIerOUERB8sf5YBQ2RWbuzp0ShXjwQ/S+bRKLncTa5Ka/DM7dnkMMtmqdK66yPPy
TgxfksDQFUhTsdEws0Pz9ySXZs7iAPaXlm4cQuHFbzG+TVW43znJvTlOTx/jfbY2
Vu4g9fcxSFZDIqmtoA/pFw1JRHUphosrphkiWhvDSksSVwRdY2Pq6FtywJWrUYxG
D+88CPc8vfZPjSgt7CJsDRsr+QJtPht1rqLel6csTrmV24NGyfid/4a++21ruesk
zu6XkzqshE2oyWOqBn1SKMBe13ycSP9EfJgdOtT/IOT62J92svslH7xT3eVlQe3C
b+BKGjrvAP6Z8PzR2/xvNFw=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = dev-Virtual-Machine
issuer=CN = aziot-edge CA dev-dev
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:DSA+SHA256:DSA+SHA384:DSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1515 bytes and written 398 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: B980356B0808C1D49172AB9A93E3C0B95AE8C2874403BFAECB303DEB961FAB5C
    Session-ID-ctx:
    Master-Key: 8A32BAD38CFF280E9D92B102BD3EF0E19B08D1080B559FADF2D2CFC06A5B5959939431203277E5F5F033277DE9D1B10D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 21 57 e4 a0 8f fa 56 4f-65 46 80 7f 6a 16 ae 70   !W....VOeF..j..p
    0010 - 45 44 93 04 34 cb 05 6f-64 42 95 69 5c e4 6b fd   ED..4..odB.i\.k.
    0020 - 84 c8 62 e8 e0 43 57 8d-da 6e c1 c3 6c 26 84 8b   ..b..CW..n..l&..
    0030 - ac ee 4d 63 21 e3 ce d7-e1 17 e3 0c 4d e4 a7 bc   ..Mc!.......M...
    0040 - 70 9a 29 96 a4 7e dc 9f-b5 ce 17 fe 52 7f dd e0   p.)..~......R...
    0050 - b4 6a 2c b0 b7 2c c3 fb-e3 0e 07 28 fc 95 ea ff   .j,..,.....(....
    0060 - f6 b2 54 bf 36 c6 71 dd-ea a5 6f 7f 9a 95 85 73   ..T.6.q...o....s
    0070 - ae 66 fa fa 96 2f c9 0b-d5 13 35 6c b9 46 c2 06   .f.../....5l.F..
    0080 - 0b 7b c4 35 a3 56 69 f2-4f f9 18 83 11 3c 01 cb   .{.5.Vi.O....<..
    0090 - 0b 79 41 66 c1 43 18 b2-e5 cd e9 fa 9f 27 ab da   .yAf.C.......'..

    Start Time: 1704957636
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes

In here is clearly visible that the tls version is 1.2. Now this is where the issue lays. Our modules are running in .net8.0 on the docker mcr.microsoft.com/dotnet/aspnet:8.0. As of this version of the docker tls lower then 1.3 is no longer supported (downgraded to security level 0).
There are other people having the same issue with databases etc resulting in the same error TLS handshake failed.
We have validated that falling back to a docker and .net version <8.0 fixes this issue (tested on .net7.0). This is however not a long term solution since both 6.0 and 7.0 will be end of support by the end of this year

Damon Barry · Answer 10 · Wed Jan 17 2024 07:20:57 GMT+0800 (China Standard Time)

@AnOdyssey I assume your modbus module is communicating with Edge Hub via the C# SDK. If so, that SDK currently only supports .NET 6.0, not .NET 8.0.

As you said, .NET 6.0 will go out of support in November 2024, but both the SDK and the IoT Edge teams are doing work right now to upgrade to .NET 8.0. Our intent is to provide the updates soon (I don't have a specific date) to give users time to upgrade to newer versions (based on .NET 8.0) before .NET 6.0 goes out support.

I'd recommend you base your module image on mcr.microsoft.com/dotnet/aspnet:6.0 for now, and see whether these problems are resolved. When IoT Edge and the C# SDK release versions that support .NET 8.0, you can upgrade your module to .NET 8.0 as well.

John Lian · Answer 11 · Wed Jan 24 2024 05:42:37 GMT+0800 (China Standard Time)

@AnOdyssey will you be able to base your module on 6.0 for now?

AnOdyssey · Answer 12 · Thu Jan 25 2024 17:45:19 GMT+0800 (China Standard Time)

Hey @jlian we have it working with 7.0

John Lian · Answer 13 · Wed Jan 31 2024 03:23:28 GMT+0800 (China Standard Time)

Ok, you rebased your module on .NET 7.0? I thought the C# SDK doesn't yet support 7.0.

Either way, are you ok for now? Will you be able to wait for our .NET 8.0 SDK fix do a patch then?

AnOdyssey · Answer 14 · Wed Jan 31 2024 03:42:53 GMT+0800 (China Standard Time)

Yes 7.0 is working without any issues so far. I can't find it anymore but I found some GitHub page where is said 7.0 is supported when I was initially looking for the supported version.

We're fine till the fix for .net 8.0 comes out.

bishal41 · Answer 15 · Thu Feb 22 2024 05:28:40 GMT+0800 (China Standard Time)

@AnOdyssey thanks. will proceed and close this issue. Keep an eye on the update IoT Edge v1.5 LTS that will use .NET 8. Thanks!

TLS handshake failed., System.IO.IOException

Expected Behavior

Current Behavior

Output of iotedge check

Device Information

Runtime Versions

Logs

Additional Information

Output of `iotedge check`