Giters

Azure / iotedge

The IoT Edge OSS project

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

edgeHub Module to Module Communication causes cleanup the Message Queue

steffenbeermann opened this issue · comments

commented

Expected Behavior

All message from the OPCPublisher should be send via the edgeHub to the custom Module.

Current Behavior

We discovered int he edgeHub log file that the task to cleanup processed and stale message for our endpoint removes message from the endpoint, see edgeHub logs from our Stage environment:

<6> 2024-01-11 00:55:23.249 +00:00 [INF] - Updated reported properties for stage-iot-edge/$edgeHub
<6> 2024-01-11 00:55:28.513 +00:00 [INF] - Updated reported properties for stage-iot-edge/$edgeHub
<6> 2024-01-11 00:57:04.045 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2024-01-11 00:57:16.755 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint stage-iot-edge/BlobStorageWriter/input1
<6> 2024-01-11 00:57:16.980 +00:00 [INF] - Cleaned up 40 messages from queue for endpoint stage-iot-edge/BlobStorageWriter/input1 and 40 messages from message store.

In our prod environment the OPCPublisher is subscribed to thousands of nodes. We see the same behavior but there the cleaned up messages are in the thousands as well:

<6> 2024-01-11 08:37:42.354 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2024-01-11 08:38:08.161 +00:00 [INF] - Cleaned up 5218 messages from queue for endpoint prod-iot-edge/BlobStorageWriter/input1 and 5218 messages from message store.

Steps to Reproduce

We could reproduce this in a fresh installation by

  1. Deploy OPCPublisher module and configure it to subscribe to some OPC Nodes (e.g. from a OPC Simulator)
  2. Configure Route to custom module
  3. Deploy custom Python Module
  4. wait some time
  5. Check Logs to find the Message Que get cleaned up

Context (Environment)

We use the Azure Edge Runtime with the following Version:

Output of iotedge check


Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ production readiness: identity certificates expiry - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
√ production readiness: Edge Hub's storage directory is persisted on the host filesystem - OK
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
36 check(s) succeeded.

Device Information

  • Tested on RedHead and Debian OS
  • Tested on arm32 and x86
  • Linux containers

Runtime Versions

  • edgeAgent: 1.4.27
  • edgeHub: 1.4.27
  • OPCPublisher: 2.9.3
  • IoTEdge: 1.4.27
  • Docker/Moby: 24.0.7-1 and API Version 1.43

Logs

We have a Active / Passive Setup for the OPCPublisher and the Custom Module. Therefor the Logs show more modules, but the issue is the same. This are the logs on startup:

aziot-edged logs 

Jan 11 10:01:14 RevPi107966 systemd[1]: Started Azure IoT Edge daemon.
Jan 11 10:01:14 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:14Z [INFO] - Starting Azure IoT Edge Daemon
Jan 11 10:01:14 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:14Z [INFO] - Version - 1.4.27
Jan 11 10:01:14 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:14Z [INFO] - Obtaining Edge device provisioning data...
Jan 11 10:01:14 RevPi107966 systemd[1]: Started Azure IoT Identity Service.
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Starting service...
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Version - 1.4.7
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Provisioning starting. Reason: Startup
Jan 11 10:01:14 RevPi107966 systemd[1]: Started Azure IoT Keys Service.
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - Starting service...
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - Version - 1.4.7
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - Loaded libaziot-keys with version 0x02010000
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - Starting server...
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 systemd[1]: Started Azure IoT Certificates Service.
Jan 11 10:01:14 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:14Z [INFO] - Starting service...
Jan 11 10:01:14 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:14Z [INFO] - Version - 1.4.7
Jan 11 10:01:14 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:14Z [INFO] - Starting server...
Jan 11 10:01:14 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:14Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
Jan 11 10:01:14 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Updated device info for axh-lab-appl-iot-gateway-revpi-02.
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Provisioning complete.
Jan 11 10:01:14 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:14Z [INFO] - Identity reconciliation started. Reason: Startup
Jan 11 10:01:14 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:14Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "355", "content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:15Z [INFO] - Hub identity "DefenderIotMicroAgent" already exists
Jan 11 10:01:15 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:15Z [INFO] - Identity reconciliation complete.
Jan 11 10:01:15 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:15Z [INFO] - Starting server...
Jan 11 10:01:15 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:15Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
Jan 11 10:01:15 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:15Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 11 10:01:15 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Device is axh-lab-appl-iot-gateway-revpi-02 on axh-lab-appl-iot-gateway-hub.azure-devices.net
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Initializing module runtime...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Using runtime network id azure-iot-edge
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Successfully initialized module runtime
Jan 11 10:01:15 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:15Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"}
Jan 11 10:01:15 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:15Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Jan 11 10:01:15 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Using existing Edge CA certificate
Jan 11 10:01:15 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:15Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Jan 11 10:01:15 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:15Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-03T08:30:06+00:00.
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping all modules...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module OPCPublisher_Passive...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module OPCPublisher_Active...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module azureblobstorageoniotedge...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module BlobStorageWriter_Passive...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module BlobStorageWriter_Active...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module edgeHub...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping module edgeAgent...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - All modules stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Detecting if device information has changed...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Device information has not changed
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting management API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module OPCPublisher_Passive
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module OPCPublisher_Active
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module azureblobstorageoniotedge
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module BlobStorageWriter_Passive
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module BlobStorageWriter_Active
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module edgeHub
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module edgeAgent
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module OPCPublisher_Passive
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module OPCPublisher_Active
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module azureblobstorageoniotedge
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting watchdog with 60 second period...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting image garbage collection task...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module BlobStorageWriter_Passive
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module BlobStorageWriter_Active
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module edgeHub
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Stopping listener for module edgeAgent
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Workload API stopped
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Watchdog checking Edge runtime status
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Edge runtime status is failed, starting module now...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting module edgeAgent...
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting new listener for module edgeAgent
Jan 11 10:01:15 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:15Z [INFO] - Starting workload API...
Jan 11 10:01:16 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:16Z [INFO] - Started Edge runtime module edgeAgent
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - <-- GET /trust-bundle?api-version=2020-07-07 {"accept": "application/json", "host": "workload.sock:80", "connection": "close"}
Jan 11 10:01:18 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:18Z [INFO] - <-- GET /certificates/aziot-edged-trust-bundle?api-version=2020-09-01 {"host": "certd.sock"}
Jan 11 10:01:18 RevPi107966 aziot-certd[18170]: 2024-01-11T10:01:18Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - Querying system info...
Jan 11 10:01:18 RevPi107966 aziot-edged[18381]: none
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - Successfully queried system info
Jan 11 10:01:18 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:18Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:19 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:19Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 11 10:01:19 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:19Z [INFO] - Querying system info...
Jan 11 10:01:19 RevPi107966 aziot-edged[18394]: none
Jan 11 10:01:19 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:19Z [INFO] - Successfully queried system info
Jan 11 10:01:19 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:19Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:20 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:20Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 11 10:01:20 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:20Z [INFO] - Querying system info...
Jan 11 10:01:20 RevPi107966 aziot-edged[18406]: none
Jan 11 10:01:20 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:20Z [INFO] - Successfully queried system info
Jan 11 10:01:20 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638388416909963673/sign?api-version=2018-06-28 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "225"}
Jan 11 10:01:21 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:21Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638388416909963673/decrypt?api-version=2022-08-03 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "5659"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "74"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "5991"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "355", "content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "736"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "748"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-identityd[18157]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - <-- POST /sign?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "966"}
Jan 11 10:01:21 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:21 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:23 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:23Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638388416909963673/encrypt?api-version=2022-08-03 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "5638"}
Jan 11 10:01:23 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:23Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "74"}
Jan 11 10:01:23 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:23 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:23Z [INFO] - <-- POST /encrypt?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "5970"}
Jan 11 10:01:23 RevPi107966 aziot-keyd[18162]: 2024-01-11T10:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:23 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - <-- POST /modules/edgeHub/stop?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - Stopping module edgeHub...
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - Stopping listener for module edgeHub
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - --> 204 {}
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - <-- POST /modules/edgeHub/start?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - Starting module edgeHub...
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - Starting new listener for module edgeHub
Jan 11 10:01:24 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:24Z [INFO] - Starting workload API...
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - --> 204 {}
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - <-- POST /modules/BlobStorageWriter_Passive/stop?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - Stopping module BlobStorageWriter_Passive...
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - Stopping listener for module BlobStorageWriter_Passive
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - --> 204 {}
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - <-- POST /modules/BlobStorageWriter_Passive/start?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - Starting module BlobStorageWriter_Passive...
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - Starting new listener for module BlobStorageWriter_Passive
Jan 11 10:01:25 RevPi107966 aziot-edged[18152]: 2024-01-11T10:01:25Z [INFO] - Starting workload API...
edge-agent logs 

2024-01-11 10:01:16  Starting Edge Agent
2024-01-11 10:01:16  Changing ownership of storage folder: /tmp/edgeAgent to 13622
2024-01-11 10:01:16  Changing ownership of backup folder: /tmp/edgeAgent_backup to 13622
2024-01-11 10:01:16  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2024-01-11 10:01:16  Completed necessary setup. Starting Edge Agent.
2024-01-11 10:01:16.714 +00:00 Edge Agent Main()
<6> 2024-01-11 10:01:17.301 +00:00 [INF] - Initializing Edge Agent.
<6> 2024-01-11 10:01:17.818 +00:00 [INF] - Version - 1.4.2.61356014 (bb9a26162c4c88b3ef9a50d33632ab78bd4247d6)
<6> 2024-01-11 10:01:17.820 +00:00 [INF] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2024-01-11 10:01:17.828 +00:00 [INF] - ModuleUpdateMode: NonBlocking
<6> 2024-01-11 10:01:17.962 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2024-01-11 10:01:18.279 +00:00 [INF] - Installing certificates [CN=aziot-edge CA axh-lab-appl-iot-gateway-revpi-02:03/21/2024 08:30:06] to Root
<6> 2024-01-11 10:01:18.756 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2024-01-11 10:01:18.802 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2024-01-11 10:01:18.819 +00:00 [INF] - Started operation Get system resources
<6> 2024-01-11 10:01:18.823 +00:00 [INF] - Collecting metadata metrics
<6> 2024-01-11 10:01:19.074 +00:00 [INF] - Set metadata metrics: 1.4.2.61356014 (bb9a26162c4c88b3ef9a50d33632ab78bd4247d6), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.4.27","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"24.0.7-1","KernelVersion":"#1 SMP PREEMPT_RT Tue, 26 Sep 2023 09:43:32 +0200","OperatingSystem":"raspbian","NumCpus":4,"TotalMemory":4023648256,"Virtualized":"no"}, True
<6> 2024-01-11 10:01:19.131 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2024-01-11 10:01:19.148 +00:00 [INF] - Started operation refresh twin config
<6> 2024-01-11 10:01:19.193 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2024-01-11 10:01:20.085 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2024-01-11 10:01:20.338 +00:00 [INF] - Started operation Metrics Scrape
<6> 2024-01-11 10:01:20.339 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2024-01-11 10:01:21.025 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2024-01-11 10:01:21.026 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2024-01-11 10:01:21.027 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2024-01-11 10:01:21.028 +00:00 [INF] - Registering request handler RestartModule
<6> 2024-01-11 10:01:22.781 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2024-01-11 10:01:23.160 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2024-01-11 10:01:23.350 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 9 and reported properties version 123.
<6> 2024-01-11 10:01:24.099 +00:00 [INF] - Plan execution started for deployment 9
<6> 2024-01-11 10:01:24.109 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module edgeHub]\n  [Start module edgeHub]\n  [Saving edgeHub to store]\n)"
<6> 2024-01-11 10:01:24.124 +00:00 [INF] - Executing command: "Stop module edgeHub"
<6> 2024-01-11 10:01:24.180 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2024-01-11 10:01:25.486 +00:00 [INF] - Executing command: "Saving edgeHub to store"
<6> 2024-01-11 10:01:25.517 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module BlobStorageWriter_Passive]\n  [Start module BlobStorageWriter_Passive]\n  [Saving BlobStorageWriter_Passive to store]\n)"
<6> 2024-01-11 10:01:25.518 +00:00 [INF] - Executing command: "Stop module BlobStorageWriter_Passive"
<6> 2024-01-11 10:01:25.529 +00:00 [INF] - Executing command: "Start module BlobStorageWriter_Passive"
<6> 2024-01-11 10:01:26.903 +00:00 [INF] - Executing command: "Saving BlobStorageWriter_Passive to store"
<6> 2024-01-11 10:01:26.917 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module OPCPublisher_Passive]\n  [Start module OPCPublisher_Passive]\n  [Saving OPCPublisher_Passive to store]\n)"
<6> 2024-01-11 10:01:26.917 +00:00 [INF] - Executing command: "Stop module OPCPublisher_Passive"
<6> 2024-01-11 10:01:26.924 +00:00 [INF] - Executing command: "Start module OPCPublisher_Passive"
<6> 2024-01-11 10:01:29.189 +00:00 [INF] - Executing command: "Saving OPCPublisher_Passive to store"
<6> 2024-01-11 10:01:29.190 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module OPCPublisher_Active]\n  [Start module OPCPublisher_Active]\n  [Saving OPCPublisher_Active to store]\n)"
<6> 2024-01-11 10:01:29.191 +00:00 [INF] - Executing command: "Stop module OPCPublisher_Active"
<6> 2024-01-11 10:01:29.203 +00:00 [INF] - Executing command: "Start module OPCPublisher_Active"
<6> 2024-01-11 10:01:30.750 +00:00 [INF] - Executing command: "Saving OPCPublisher_Active to store"
<6> 2024-01-11 10:01:30.751 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module BlobStorageWriter_Active]\n  [Start module BlobStorageWriter_Active]\n  [Saving BlobStorageWriter_Active to store]\n)"
<6> 2024-01-11 10:01:30.751 +00:00 [INF] - Executing command: "Stop module BlobStorageWriter_Active"
<6> 2024-01-11 10:01:30.755 +00:00 [INF] - Executing command: "Start module BlobStorageWriter_Active"
<6> 2024-01-11 10:01:32.397 +00:00 [INF] - Executing command: "Saving BlobStorageWriter_Active to store"
<6> 2024-01-11 10:01:32.401 +00:00 [INF] - Executing command: "Command Group: (\n  [Stop module azureblobstorageoniotedge]\n  [Start module azureblobstorageoniotedge]\n  [Saving azureblobstorageoniotedge to store]\n)"
<6> 2024-01-11 10:01:32.402 +00:00 [INF] - Executing command: "Stop module azureblobstorageoniotedge"
<6> 2024-01-11 10:01:32.406 +00:00 [INF] - Executing command: "Start module azureblobstorageoniotedge"
<6> 2024-01-11 10:01:34.800 +00:00 [INF] - Executing command: "Saving azureblobstorageoniotedge to store"
<6> 2024-01-11 10:01:34.802 +00:00 [INF] - Plan execution ended for deployment 9
<6> 2024-01-11 10:01:35.600 +00:00 [INF] - Updated reported properties
<6> 2024-01-11 10:01:41.094 +00:00 [INF] - Updated reported properties
edge-hub logs 
2024-01-11 10:01:25  Starting Edge Hub
2024-01-11 10:01:25  Starting Edge Hub
2024-01-11 10:01:25  Changing ownership of storage folder: /tmp/edgeHub to 13623
2024-01-11 10:01:25  Changing ownership of backup folder: /tmp/edgeHub_backup to 13623
2024-01-11 10:01:25.763 +00:00 Edge Hub Main()
<6> 2024-01-11 10:01:30.081 +00:00 [INF] - Installing certificates [CN=aziot-edge CA axh-lab-appl-iot-gateway-revpi-02:03/21/2024 08:30:06] to Root
<6> 2024-01-11 10:01:30.313 +00:00 [INF] - Installing certificates [CN=aziot-edge CA axh-lab-appl-iot-gateway-revpi-02:03/21/2024 08:30:06] to Root
<6> 2024-01-11 10:01:30.338 +00:00 [INF] - Enabling SSL protocols: Tls12
<6> 2024-01-11 10:01:30.706 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2024-01-11 10:01:33.192 +00:00 [INF] - Created persistent store at /tmp/edgeHub
<6> 2024-01-11 10:01:33.578 +00:00 [INF] - Initializing Edge Hub
<6> 2024-01-11 10:01:33.581 +00:00 [INF] - 
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2024-01-11 10:01:33.599 +00:00 [INF] - Version - 1.4.2.61356014 (bb9a26162c4c88b3ef9a50d33632ab78bd4247d6)
<6> 2024-01-11 10:01:33.601 +00:00 [INF] - OptimizeForPerformance=False
<6> 2024-01-11 10:01:33.602 +00:00 [INF] - MessageAckTimeoutSecs=30
<6> 2024-01-11 10:01:33.615 +00:00 [INF] - Loaded server certificate with expiration date of "2024-02-10T10:01:29.0000000+00:00"
<6> 2024-01-11 10:01:33.731 +00:00 [INF] - Using Asp Net server for metrics
<6> 2024-01-11 10:01:34.100 +00:00 [INF] - Created new message store
<6> 2024-01-11 10:01:34.101 +00:00 [INF] - Started task to cleanup processed and stale messages
<6> 2024-01-11 10:01:34.688 +00:00 [INF] - Created DeviceConnectivityManager with connected check frequency 00:05:00 and disconnected check frequency 00:02:00
<6> 2024-01-11 10:01:35.611 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02
<6> 2024-01-11 10:01:35.619 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/$edgeAgent
<6> 2024-01-11 10:01:35.619 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:35.619 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:35.620 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:01:35.620 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/DefenderIotMicroAgent
<6> 2024-01-11 10:01:35.620 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:35.620 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:35.621 +00:00 [INF] - Add node: axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:35.627 +00:00 [INF] - Created device scope identities cache
<6> 2024-01-11 10:01:35.667 +00:00 [INF] - Initialized storing twin manager
<6> 2024-01-11 10:01:35.695 +00:00 [INF] - Updated node: axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:35.696 +00:00 [INF] - Initializing configuration
<6> 2024-01-11 10:01:35.720 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2024-01-11 10:01:35.739 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:35.789 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/$edgeHub connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:01:36.080 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/$edgeHub via AMQP...
<6> 2024-01-11 10:01:36.253 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:38.518 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false,"DisableConnectivityCheck":false}
<6> 2024-01-11 10:01:38.698 +00:00 [INF] - Starting timer to authenticate connections with a period of 300 seconds
<6> 2024-01-11 10:01:39.390 +00:00 [INF] - Scheduling server certificate renewal for "2024-02-10T09:58:59.0004013Z".
<6> 2024-01-11 10:01:41.447 +00:00 [INF] - Set the following 2 route(s) in edge hub
<6> 2024-01-11 10:01:41.451 +00:00 [INF] - fileupload_opc_active: FROM //modules/OPCPublisher_Active INTO BrokeredEndpoint("/modules/BlobStorageWriter_Active/inputs/opcua")
<6> 2024-01-11 10:01:41.451 +00:00 [INF] - fileupload_opc_passive: FROM /messages/modules/OPCPublisher_Passive INTO BrokeredEndpoint("/modules/BlobStorageWriter_Passive/inputs/opcua")
<6> 2024-01-11 10:01:41.471 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2024-01-11 10:01:41.479 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2024-01-11 10:01:41.696 +00:00 [INF] - Started operation Get EdgeHub config
<6> 2024-01-11 10:01:41.719 +00:00 [INF] - Initialized edge hub configuration
<6> 2024-01-11 10:01:41.741 +00:00 [INF] - Starting protocol heads - (MQTT, AMQP, HTTP)
<6> 2024-01-11 10:01:41.785 +00:00 [INF] - Starting MQTT head
<6> 2024-01-11 10:01:41.843 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02
<6> 2024-01-11 10:01:41.848 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/$edgeAgent
<6> 2024-01-11 10:01:41.848 +00:00 [INF] - Updated node: axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:42.114 +00:00 [INF] - Service identity for axh-lab-appl-iot-gateway-revpi-02/$edgeHub in device scope was updated.
<6> 2024-01-11 10:01:42.116 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/DefenderIotMicroAgent
<6> 2024-01-11 10:01:42.117 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:42.118 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:42.118 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:42.119 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:01:42.120 +00:00 [INF] - Not changed node: axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:42.134 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes.
<6> 2024-01-11 10:01:42.166 +00:00 [INF] - Initializing TLS endpoint on port 8883 for MQTT head.
<6> 2024-01-11 10:01:42.413 +00:00 [INF] - Starting AMQP head
<6> 2024-01-11 10:01:42.429 +00:00 [INF] - Started MQTT head
<6> 2024-01-11 10:01:42.568 +00:00 [INF] - Exiting disconnected state
<6> 2024-01-11 10:01:42.974 +00:00 [INF] - Started AMQP head
<6> 2024-01-11 10:01:42.976 +00:00 [INF] - Received device connected callback
<6> 2024-01-11 10:01:43.000 +00:00 [INF] - Starting HTTP head
<6> 2024-01-11 10:01:43.004 +00:00 [INF] - Device connected to cloud, processing subscriptions for connected clients.
<6> 2024-01-11 10:01:43.015 +00:00 [INF] - Processing subscriptions for client axh-lab-appl-iot-gateway-revpi-02/$edgeHub on device connected to cloud.
<6> 2024-01-11 10:01:43.036 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/$edgeHub for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:43.050 +00:00 [INF] - Skipping cache refresh, waiting 119 seconds until refreshing again.
<6> 2024-01-11 10:01:43.147 +00:00 [INF] - Entering connected state
<6> 2024-01-11 10:01:43.197 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/$edgeHub is True
<6> 2024-01-11 10:01:43.236 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/$edgeHub changed to ConnectionEstablished
<6> 2024-01-11 10:01:43.243 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/$edgeHub connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:01:43.244 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/$edgeHub for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:43.256 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/$edgeHub via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:01:43.275 +00:00 [INF] - Initialized cloud proxy 273e319b-9ca2-4bcb-9bd7-59468d1178f7 for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:43.282 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:44.689 +00:00 [INF] - Updated reported properties for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:44.959 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<4> 2024-01-11 10:01:45.303 +00:00 [WRN] - Overriding address(es) '"http://+:80"'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
<6> 2024-01-11 10:01:45.387 +00:00 [INF] - Started HTTP head
<6> 2024-01-11 10:01:45.954 +00:00 [INF] - Obtained edge hub config from module twin
<6> 2024-01-11 10:01:46.199 +00:00 [INF] - Set the following 2 route(s) in edge hub
<6> 2024-01-11 10:01:46.199 +00:00 [INF] - fileupload_opc_active: FROM /messages/modules/OPCPublisher_Active INTO BrokeredEndpoint("/modules/BlobStorageWriter_Active/inputs/opcua")
<6> 2024-01-11 10:01:46.200 +00:00 [INF] - fileupload_opc_passive: FROM /messages/modules/OPCPublisher_Passive INTO BrokeredEndpoint("/modules/BlobStorageWriter_Passive/inputs/opcua")
<6> 2024-01-11 10:01:46.200 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2024-01-11 10:01:46.201 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2024-01-11 10:01:46.280 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge in device scope authenticated locally.
<6> 2024-01-11 10:01:46.314 +00:00 [INF] - Successfully generated identity for clientId axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge and username RevPi107966/axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge/?api-version=2020-09-30&DeviceClientType=.NET%2F1.41.3%20%28.NET%206.0.18%3B%20Linux%205.10.152-rt75-v7l%20%231%20SMP%20PREEMPT_RT%20Tue%2C%2026%20Sep%202023%2009%3A43%3A32%20%2B0200%3B%20Arm%29
<6> 2024-01-11 10:01:46.332 +00:00 [INF] - "ClientAuthenticated, axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge, 2cc1790c"
<6> 2024-01-11 10:01:46.440 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.443 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:01:46.513 +00:00 [INF] - Bind device proxy for device axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.516 +00:00 [INF] - Binding message channel for device Id axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.533 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.543 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge via AMQP...
<6> 2024-01-11 10:01:46.692 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates for client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge.
<6> 2024-01-11 10:01:46.726 +00:00 [INF] - Set subscriptions from session state for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.837 +00:00 [INF] - Processing subscriptions TwinResponse, DesiredPropertyUpdates for client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge.
<6> 2024-01-11 10:01:46.844 +00:00 [INF] - Set subscriptions from session state for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.852 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge is True
<6> 2024-01-11 10:01:46.852 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge changed to ConnectionEstablished
<6> 2024-01-11 10:01:46.852 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:01:46.853 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:46.853 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:01:46.853 +00:00 [INF] - Initialized cloud proxy 45bc4617-fee2-4d82-80b9-eb6e0a1f5951 for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:46.854 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:47.127 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:47.128 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:47.128 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:47.434 +00:00 [INF] - New token received on the Cbs link
<6> 2024-01-11 10:01:47.919 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive in device scope authenticated locally.
<6> 2024-01-11 10:01:47.943 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:47.944 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:01:47.946 +00:00 [INF] - Bind device proxy for device axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:47.948 +00:00 [INF] - Initialized device listener in the AMQP protocol head for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:47.948 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive via AMQP...
<6> 2024-01-11 10:01:47.980 +00:00 [INF] - Opened link Events for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.199 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive is True
<6> 2024-01-11 10:01:48.199 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive changed to ConnectionEstablished
<6> 2024-01-11 10:01:48.199 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:01:48.200 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:48.200 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:01:48.201 +00:00 [INF] - Initialized cloud proxy 08f7a6e8-d3c4-459a-b7ab-d50fb11465e4 for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.202 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.323 +00:00 [INF] - Opened link TwinReceiving for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.335 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.338 +00:00 [INF] - Opened link TwinSending for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.536 +00:00 [INF] - Opened link MethodReceiving for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.598 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:48.597 +00:00 [INF] - Opened link MethodSending for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Passive
<6> 2024-01-11 10:01:49.296 +00:00 [INF] - New token received on the Cbs link
<6> 2024-01-11 10:01:49.708 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active in device scope authenticated locally.
<6> 2024-01-11 10:01:49.709 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:49.709 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:01:49.711 +00:00 [INF] - Bind device proxy for device axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:49.719 +00:00 [INF] - Initialized device listener in the AMQP protocol head for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:49.719 +00:00 [INF] - Opened link Events for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:49.721 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active via AMQP...
<6> 2024-01-11 10:01:49.978 +00:00 [INF] - Opened link TwinReceiving for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.008 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active is True
<6> 2024-01-11 10:01:50.008 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active changed to ConnectionEstablished
<6> 2024-01-11 10:01:50.008 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:01:50.008 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:50.008 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:01:50.009 +00:00 [INF] - Initialized cloud proxy 5655a154-0003-4efb-a723-ec9e9f760895 for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.009 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.092 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.092 +00:00 [INF] - Opened link TwinSending for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.273 +00:00 [INF] - Opened link MethodReceiving for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.369 +00:00 [INF] - Opened link MethodSending for axh-lab-appl-iot-gateway-revpi-02/OPCPublisher_Active
<6> 2024-01-11 10:01:50.710 +00:00 [INF] - Updated reported properties for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:01:52.357 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active in device scope authenticated locally.
<6> 2024-01-11 10:01:52.359 +00:00 [INF] - Successfully generated identity for clientId axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active and username axh-lab-appl-iot-gateway-hub.azure-devices.net/axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.12.0%283.9.17%3BLinux%20%231%20SMP%20PREEMPT_RT%20Tue%2C%2026%20Sep%202023%2009%3A43%3A32%20%2B0200%3Barmv7l%29
<6> 2024-01-11 10:01:52.360 +00:00 [INF] - "ClientAuthenticated, axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active, 6f65b038"
<6> 2024-01-11 10:01:52.394 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.394 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:01:52.398 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active via AMQP...
<6> 2024-01-11 10:01:52.410 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.411 +00:00 [INF] - Bind device proxy for device axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.411 +00:00 [INF] - Binding message channel for device Id axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.412 +00:00 [INF] - Processing subscriptions ModuleMessages for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active.
<6> 2024-01-11 10:01:52.414 +00:00 [INF] - Set subscriptions from session state for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.629 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active is True
<6> 2024-01-11 10:01:52.629 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active changed to ConnectionEstablished
<6> 2024-01-11 10:01:52.630 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:01:52.630 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:01:52.631 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:01:52.631 +00:00 [INF] - Initialized cloud proxy 98f5717c-39ff-4fa6-bd0c-d37be171946d for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.632 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:52.671 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Active
<6> 2024-01-11 10:01:56.200 +00:00 [INF] - Updated reported properties for axh-lab-appl-iot-gateway-revpi-02/azureblobstorageoniotedge
<6> 2024-01-11 10:01:56.696 +00:00 [INF] - Updated reported properties for axh-lab-appl-iot-gateway-revpi-02/$edgeHub
<6> 2024-01-11 10:02:11.880 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive in device scope authenticated locally.
<6> 2024-01-11 10:02:11.882 +00:00 [INF] - Successfully generated identity for clientId axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive and username axh-lab-appl-iot-gateway-hub.azure-devices.net/axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive/?api-version=2019-10-01&DeviceClientType=azure-iot-device-iothub-py%2F2.12.0%283.9.17%3BLinux%20%231%20SMP%20PREEMPT_RT%20Tue%2C%2026%20Sep%202023%2009%3A43%3A32%20%2B0200%3Barmv7l%29
<6> 2024-01-11 10:02:11.883 +00:00 [INF] - "ClientAuthenticated, axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive, 0e7186a6"
<6> 2024-01-11 10:02:11.894 +00:00 [INF] - New device connection for device axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:11.896 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive connected to edgeHub, processing existing subscriptions.
<6> 2024-01-11 10:02:11.899 +00:00 [INF] - Attempting to connect to IoT Hub for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive via AMQP...
<6> 2024-01-11 10:02:11.913 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:11.914 +00:00 [INF] - Bind device proxy for device axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:11.914 +00:00 [INF] - Binding message channel for device Id axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:11.915 +00:00 [INF] - Processing subscriptions ModuleMessages for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive.
<6> 2024-01-11 10:02:11.918 +00:00 [INF] - Set subscriptions from session state for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:12.121 +00:00 [INF] - Cloud connection for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive is True
<6> 2024-01-11 10:02:12.121 +00:00 [INF] - Connection status for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive changed to ConnectionEstablished
<6> 2024-01-11 10:02:12.121 +00:00 [INF] - Client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive connected to cloud, processing existing subscriptions.
<6> 2024-01-11 10:02:12.121 +00:00 [INF] - Skipping axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive for subscription processing, as it is currently being processed.
<6> 2024-01-11 10:02:12.122 +00:00 [INF] - Created cloud proxy for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive via AMQP, with client operation timeout 20 seconds.
<6> 2024-01-11 10:02:12.122 +00:00 [INF] - Initialized cloud proxy b2f5e2cc-c4e9-477d-aa9d-faf0179e9f26 for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:12.122 +00:00 [INF] - Created cloud connection for client axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:12.124 +00:00 [INF] - Processing pending subscriptions for axh-lab-appl-iot-gateway-revpi-02/BlobStorageWriter_Passive
<6> 2024-01-11 10:02:17.230 +00:00 [INF] - Updated reported properties for axh-lab-appl-iot-gateway-revpi-02/$edgeHub

.... 

here a snippet with the cleanup

<6> 2024-01-11 08:53:56.645 +00:00 [INF] - fileupload: FROM /messages/modules/OPCPublisher INTO BrokeredEndpoint("/modules/BlobStorageWriter/inputs/input1")
<6> 2024-01-11 08:53:56.645 +00:00 [INF] - Updated message store TTL to 7200 seconds
<6> 2024-01-11 08:53:56.645 +00:00 [INF] - Updated the edge hub store and forward configuration
<6> 2024-01-11 08:53:56.645 +00:00 [INF] - Successfully completed periodic operation Get EdgeHub config
<6> 2024-01-11 08:53:56.821 +00:00 [INF] - Updated reported properties for prod-iot-edge/$edgeHub
<6> 2024-01-11 08:57:42.345 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
last UploadModuleLogs at: 09:20
<6> 2024-01-11 09:02:42.333 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2024-01-11 09:03:24.699 +00:00 [INF] - Starting refresh of device scope identities cache
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Not changed node: prod-iot-edge
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Not changed node: prod-iot-edge/$edgeAgent
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Not changed node: prod-iot-edge/$edgeHub
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Not changed node: prod-iot-edge/OPCPublisher
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Not changed node: prod-iot-edge/BlobStorageWriter
<6> 2024-01-11 09:03:25.897 +00:00 [INF] - Done refreshing device scope identities cache. Waiting for 60 minutes.
<6> 2024-01-11 09:07:42.334 +00:00 [INF] - Entering periodic task to reauthenticate connected clients
<6> 2024-01-11 09:08:08.162 +00:00 [INF] - Started task to cleanup processed and stale messages for endpoint prod-iot-edge/BlobStorageWriter/input1
<6> 2024-01-11 09:08:47.558 +00:00 [INF] - Cleaned up 5216 messages from queue for endpoint prod-iot-edge/BlobStorageWriter/input1 and 5216 messages from message store.
<6> 2024-01-11 09:08:58.056 +00:00 [INF] - "Closing connection for device: prod-iot-edge/BlobStorageWriter, , "
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Disposing MessagingServiceClient for device Id prod-iot-edge/BlobStorageWriter because of exception - 
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Setting device proxy inactive for device Id prod-iot-edge/BlobStorageWriter
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Removing device connection for device prod-iot-edge/BlobStorageWriter with removeCloudConnection flag 'True'.
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Closing receiver in cloud proxy 0c0dc127-4916-4866-a09b-31e50c687007 for prod-iot-edge/BlobStorageWriter
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Closed cloud proxy 0c0dc127-4916-4866-a09b-31e50c687007 for prod-iot-edge/BlobStorageWriter
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Device connection removed for device prod-iot-edge/BlobStorageWriter
<6> 2024-01-11 09:08:58.057 +00:00 [INF] - Remove device connection for device prod-iot-edge/BlobStorageWriter
commented

@steffenbeermann do you see messages actually gone missing?

The way edgeHub works is that:

  • when a message arrives, it puts that message in a queue
  • then when it has a chance (the device is online) it sends messages out from that queue
  • however, it does not remove the message from the queue when it sends out
  • instead, it maintains a pointer which shows what message was sent out last time
  • then there is a job that runs periodically (I believe it is 30mins by default)
  • This job does two things
    • it removes messages from the queue up to the index that points to the last sent message
    • it also removes all expired messages (even if it hasn't been sent out). The expiry time is 2 hours I believe by default

So the clean up message you see is normal. It means only that since the last cleaning, there were xy messages put into the queue, and it found so many to clean up to the pointer.

Let me know if you have messages missing or you need further explanation

commented

Thank you for the explanation. I would have one follow up questions:
How do we now, if messages are not delivered correctly? Would there be a different log for that?

commented

@steffenbeermann from the logs you cannot see the reason why a message has been cleaned up (it has been sent out or expired). However, if you use metrics, then querying the "edgehub_messages_dropped_total" counter with the "reason" dimension, there is a value for reason as "ttl_expiry". Those are the unsent messages that got cleaned because those were too old.

There are a few reasons for an m2m message to get stuck:

  • the receiving module is too slow. In this case the message queue starts growing, and after a while messages in the queue get too old. You don't see an error message about this, however, you can monitor the queue length using metrics
  • the receiving module does not acknowledge the messages. You would see error messages about this in the log. These messages don't get drop (before they expire), but edge tries to resend them.
  • the receiving module is not online (is not connected to edge to receive messages). In this case the messages are waiting in the message queue, and they expire after a while.

If you worry about messages being stuck or slow processing time, I suggest setting up metrics.

commented

Okay, perfect. Than you very much for the information. That would solve our issue / misunderstanding.