edgeAgent keep restarting after [ERR] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Encrypt

Question

edgeAgent keep restarting after [ERR] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Encrypt

guy-radford-sunswap opened this issue 4 months ago · comments

guy-radford-sunswap commented 4 months ago

Expected Behavior

Edge module should be able to run without the edgeAgent filing

Current Behavior

edgeAgent failed with error: [ERR] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Encrypt

Seee edgeAgent logs below for the error.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

This appeared on a system that had been running for a while and on a newly commissioned device.
On the previously running device, this appears around 15:00 on the 17th Jan GMT.

I use the same Disk image for our development environment and as yet have not seen this, but I am still investigating and will update as new information comes to light.

Context (Environment)

Output of `iotedge check`

Click here


root@ss-ped-dv1:~# iotedge check

Configuration checks (aziot-identity-service)
---------------------------------------------
√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)
--------------------------------------------
√ host can connect to and perform TLS handshake with iothub AMQP port - OK
√ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with iothub MQTT port - OK

Configuration checks
--------------------
√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
√ configuration has correct URIs for daemon mgmt endpoint - OK
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
√ DNS server - OK
√ production readiness: logs policy - OK
√ production readiness: Edge Agent's storage directory is persisted on the host filesystem - OK
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
    Could not check current state of edgeHub container
√ Agent image is valid and can be pulled from upstream - OK
√ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - OK

Connectivity checks
-------------------
√ container on the default network can connect to upstream AMQP port - OK
√ container on the default network can connect to upstream HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to upstream AMQP port - OK
√ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
34 check(s) succeeded.
1 check(s) raised errors. Re-run with --verbose for more details.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Device Information

Host OS: Raspbian GNU/Linux 11 (bullseye) 32bit
Architecture: arm64
Container OS: Linux containers

Runtime Versions

aziot-edged [run iotedge version]: iotedge 1.4.27
Edge Agent [image tag (e.g. 1.0.0)]: 1.4 (9d261383b715)
Edge Hub [image tag (e.g. 1.0.0)]: 1.4 (b0b3709fa3fb)
Docker/Moby [run docker version]: 24.0.7-1

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs

In comment below...

edge-agent logs


2024-01-18 09:18:45  Starting Edge Agent
2024-01-18 09:18:45  Creating UID 13622 as edgeagentuser
2024-01-18 09:18:45  Changing ownership of storage folder: /tmp/edgeAgent/edgeAgent to 13622
2024-01-18 09:18:45  Creating backup folder: /tmp/edgeAgent_backup
2024-01-18 09:18:45  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2024-01-18 09:18:45  Completed necessary setup. Starting Edge Agent.
2024-01-18 09:18:45.285 +00:00 Edge Agent Main()
<6> 2024-01-18 09:18:45.860 +00:00 [INF] - Initializing Edge Agent.
<6> 2024-01-18 09:18:46.350 +00:00 [INF] - Version - 1.4.29.85541525 (f049017a5072f85aec250e945ebb2ca52e4af3f4)
<6> 2024-01-18 09:18:46.352 +00:00 [INF] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2024-01-18 09:18:46.362 +00:00 [INF] - ModuleUpdateMode: NonBlocking
<6> 2024-01-18 09:18:46.571 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2024-01-18 09:18:46.958 +00:00 [INF] - Installing certificates [CN=aziot-edge CA ss-ped-dv1:04/15/2024 11:28:57] to Root
<6> 2024-01-18 09:18:47.462 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2024-01-18 09:18:47.509 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2024-01-18 09:18:47.526 +00:00 [INF] - Started operation Get system resources
<6> 2024-01-18 09:18:47.531 +00:00 [INF] - Collecting metadata metrics
<6> 2024-01-18 09:18:47.781 +00:00 [INF] - Set metadata metrics: 1.4.29.85541525 (f049017a5072f85aec250e945ebb2ca52e4af3f4), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.4.27","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"24.0.7-1","KernelVersion":"#1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023","OperatingSystem":"raspbian","NumCpus":4,"TotalMemory":953647104,"Virtualized":"no"}, True
<6> 2024-01-18 09:18:47.839 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2024-01-18 09:18:47.853 +00:00 [INF] - Started operation refresh twin config
<6> 2024-01-18 09:18:47.902 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2024-01-18 09:18:48.811 +00:00 [INF] - Created persistent store at /tmp/edgeAgent/edgeAgent
<6> 2024-01-18 09:18:49.074 +00:00 [INF] - Started operation Metrics Scrape
<6> 2024-01-18 09:18:49.080 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2024-01-18 09:18:50.255 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2024-01-18 09:18:50.256 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2024-01-18 09:18:50.257 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2024-01-18 09:18:50.258 +00:00 [INF] - Registering request handler RestartModule
<6> 2024-01-18 09:18:50.904 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2024-01-18 09:18:51.253 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2024-01-18 09:18:51.421 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 9 and reported properties version 6047.
<6> 2024-01-18 09:18:59.529 +00:00 [INF] - Plan execution started for deployment 9
<6> 2024-01-18 09:18:59.538 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.550 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.551 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.551 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.552 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.552 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.553 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.553 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.553 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.554 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.554 +00:00 [INF] - Executing command: "[Null]"
<6> 2024-01-18 09:18:59.592 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare module endurance_gps]\n  [Create module endurance_gps]\n)]\n  [Start module endurance_gps]\n)"
<6> 2024-01-18 09:18:59.600 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare module endurance_gps]\n  [Create module endurance_gps]\n)"
<6> 2024-01-18 09:19:02.614 +00:00 [INF] - Executing command: "Start module endurance_gps"
<6> 2024-01-18 09:19:02.938 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare module endurance_lte]\n  [Create module endurance_lte]\n)]\n  [Start module endurance_lte]\n)"
<6> 2024-01-18 09:19:02.943 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare module endurance_lte]\n  [Create module endurance_lte]\n)"
<6> 2024-01-18 09:19:05.336 +00:00 [INF] - Executing command: "Start module endurance_lte"
<6> 2024-01-18 09:19:06.022 +00:00 [INF] - Executing command: "Command Group: (\n  [Command Group: (\n  [Prepare module edgeHub]\n  [Create module edgeHub]\n)]\n  [Start module edgeHub]\n)"
<6> 2024-01-18 09:19:06.023 +00:00 [INF] - Executing command: "Command Group: (\n  [Prepare module edgeHub]\n  [Create module edgeHub]\n)"
<6> 2024-01-18 09:19:12.400 +00:00 [INF] - Termination requested, initiating shutdown.
<6> 2024-01-18 09:19:12.400 +00:00 [INF] - Waiting for cleanup to finish
<6> 2024-01-18 09:19:13.752 +00:00 [INF] - Plan execution for deployment 9 was cancelled
<6> 2024-01-18 09:19:13.754 +00:00 [INF] - Plan execution ended for deployment 9
<3> 2024-01-18 09:19:13.816 +00:00 [ERR] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Encrypt
System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 30
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 229
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 185
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.<>c__DisplayClass5_0.<EncryptAsync>b__0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 66
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.AsyncExecution`1.ExecuteAsyncImpl(Task ignore) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/AsyncExecution[T].cs:line 64
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.AsyncExecution`1.ExecuteAsync() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/AsyncExecution[T].cs:line 43
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.RetryPolicy.ExecuteAsync[TResult](Func`1 taskFunc, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/RetryPolicy.cs:line 254
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.RetryPolicy.ExecuteAsync[TResult](Func`1 taskFunc) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/RetryPolicy.cs:line 233
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.ExecuteWithRetry[T](Func`1 func, Action`1 onRetry, ITransientErrorDetectionStrategy transientErrorDetection) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 89
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.EncryptAsync(String initializationVector, String plainText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 66
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.EncryptAsync(String initializationVector, String plainText)
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.EncryptAsync(String initializationVector, String plainText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClient.cs:line 28
   at Microsoft.Azure.Devices.Edge.Util.Edged.EncryptionProvider.EncryptAsync(String plainText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/EncryptionProvider.cs:line 43
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.UpdateCurrentConfig(DeploymentConfigInfo deploymentConfigInfo) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 312
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.UpdateCurrentConfig(DeploymentConfigInfo deploymentConfigInfo)
   at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 163
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at Microsoft.Azure.Devices.Edge.Agent.Core.PlanRunner.OrderedRetryPlanRunner.ExecuteAsync(Int64 deploymentId, Plan plan, CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/planrunner/OrdererdRetryPlanRunner.cs:line 132
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 37
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.Azure.Devices.Edge.Agent.Core.LoggingCommandFactory.LoggingCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/LoggingCommandFactory.cs:line 69
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at Microsoft.Azure.Devices.Edge.Agent.Core.Commands.GroupCommand.ExecuteAsync(CancellationToken token) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/commands/GroupCommand.cs:line 37
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.Throttle[T](Func`1 identityOperation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 164
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient.<>c__DisplayClass26_0.<<Throttle>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/ModuleManagementHttpClient.cs:line 145
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.ModuleManagemlient.CreateModuleAsync(ModuleSpec moduleSpec) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/ModuleManagementHttpClient.cs:line 102
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 171
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessInnerTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessCompletedOuterTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.RunOrQueueCompletionAction(ITaskCompletionAction completionAction, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishStageTwo()
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(Task task, Boolean taskWasPreviouslyQueued)
   at System.Threading.Tasks.TaskScheduler.TryRunInline(Task task, Boolean taskWasPreviouslyQueued)
   at System.Threading.Tasks.TaskContinuation.InlineIfPossibleOrElseQueue(Task task, Boolean needsProtection)
   at System.Threading.Tasks.ContinueWithTaskContinuation.Run(Task completedTask, Boolean canInlineContinuationTask)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.GeneratedCode.EdgeletHttpClient.CreateModuleAsync(String api_version, ModuleSpec module, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/generatedCode/EdgeletHttpClient.cs:line 252
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 48
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.DeserializeResponse(HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 69
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.SetResponseStatusLine(HttpResponseMessage httpResponse, HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 158
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpBufferedStream.ReadLineAsync(CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpBufferedStream.cs:line 78
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.IO.BufferedStream.ReadFromUnderlyingStreamAsync(Memory`1 buffer, CancellationToken cancellationToken, Int32 bytesAlreadySatisfied, Task semaphoreLockTask)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Threading.ThreadPool.<>c.<.cctor>b__86_0(Object state)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
   at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal()
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 48
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.EncryptAsync(String api_version, String name, String genid, EncryptRequest payload, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 229
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 135
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 61

edge-hub logs


None as does not start

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

guy-radford-sunswap · Answer 1 · Thu Jan 18 2024 17:44:53 GMT+0800 (China Standard Time)

aziot-edged logs



Jan 18 09:20:42 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:42Z [INFO] - Pulling image via tag sunswapcr.azurecr.io/endurance-edge/endurance_lte:v1.0.3...
Jan 18 09:20:43 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:43Z [INFO] - Successfully pulled image sunswapcr.azurecr.io/endurance-edge/endurance_lte:v1.0.3
Jan 18 09:20:43 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:43Z [INFO] - --> 204 {}
Jan 18 09:20:43 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:43Z [INFO] - <-- POST /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "1187"}
Jan 18 09:20:44 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:44Z [INFO] - Pulling image via tag sunswapcr.azurecr.io/endurance-edge/endurance_lte:v1.0.3...
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Successfully pulled image sunswapcr.azurecr.io/endurance-edge/endurance_lte:v1.0.3
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Creating module endurance_lte...
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Creating image via tag sunswapcr.azurecr.io/endurance-edge/endurance_lte:v1.0.3...
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - --> 201 {"content-type": "application/json"}
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - <-- POST /modules/endurance_lte/start?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Starting module endurance_lte...
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Starting new listener for module endurance_lte
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - Starting workload API...
Jan 18 09:20:45 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:45Z [INFO] - --> 204 {}
Jan 18 09:20:46 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:46Z [INFO] - <-- POST /modules/endurance_cs_setup/prepareupdate?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "543"}
Jan 18 09:20:46 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:46Z [INFO] - Pulling image via tag sunswapcr.azurecr.io/endurance-edge/endurance_cs_setup:v1.5.2...
Jan 18 09:20:47 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:47Z [INFO] - Successfully pulled image sunswapcr.azurecr.io/endurance-edge/endurance_cs_setup:v1.5.2
Jan 18 09:20:47 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:47Z [INFO] - --> 204 {}
Jan 18 09:20:47 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:47Z [INFO] - <-- POST /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "1117"}
Jan 18 09:20:47 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:47Z [INFO] - Pulling image via tag sunswapcr.azurecr.io/endurance-edge/endurance_cs_setup:v1.5.2...
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - <-- GET /trust-bundle?api-version=2022-08-03 {"host": "localhost", "user-agent": "azure-iot-device-iothub-py%2F2.12.0%283.8.17%3BLinux+%231642+SMP+PREEMPT+Mon+Apr++3+17%3A24%3A16+BST+2023%3Baarch64%29", "accept-encoding": "gzip, deflate", "accept": "*/*", "connection": "keep-alive"}
Jan 18 09:20:48 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:20:48Z [INFO] - <-- GET /certificates/aziot-edged-trust-bundle?api-version=2020-09-01 {"host": "certd.sock"}
Jan 18 09:20:48 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - <-- POST /modules/endurance_lte/genid/638410146821301662/sign?api-version=2022-08-03 {"host": "localhost", "user-agent": "azure-iot-device-iothub-py%2F2.12.0%283.8.17%3BLinux%20%231642%20SMP%20PREEMPT%20Mon%20Apr%20%203%2017%3A24%3A16%20BST%202023%3Baarch64%29", "accept-encoding": "gzip, deflate", "accept": "*/*", "connection": "keep-alive", "content-length": "198"}
Jan 18 09:20:48 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:20:48Z [INFO] - <-- GET /identities/modules/endurance_lte?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "344"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "752"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "344"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "760"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - <-- POST /sign?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "950"}
Jan 18 09:20:48 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - Successfully pulled image sunswapcr.azurecr.io/endurance-edge/endurance_cs_setup:v1.5.2
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - Creating module endurance_cs_setup...
Jan 18 09:20:48 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:48Z [INFO] - Creating image via tag sunswapcr.azurecr.io/endurance-edge/endurance_cs_setup:v1.5.2...
Jan 18 09:20:49 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:49Z [INFO] - Watchdog checking Edge runtime status
Jan 18 09:20:49 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:49Z [INFO] - Edge runtime is running
Jan 18 09:20:49 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:49Z [INFO] - <-- POST /device/reprovision?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - --> 201 {"content-type": "application/json"}
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - <-- POST /modules/endurance_cs_setup/start?api-version=2022-08-03 {"host": "mgmt.sock:80", "connection": "close", "content-type": "application/json; charset=utf-8", "content-length": "0"}
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Edge daemon will reprovision and restart
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Watchdog stopped
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping all modules...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - --> 204 {}
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Starting module endurance_cs_setup...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Starting new listener for module endurance_cs_setup
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Starting workload API...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping module endurance_cs_setup...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping module endurance_lte...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping module edgeAgent...
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping listener for module endurance_cs_setup
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping listener for module endurance_lte
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Stopping listener for module edgeAgent
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Workload API stopped
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Workload API stopped
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - Workload API stopped
Jan 18 09:20:51 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:20:51Z [INFO] - --> 204 {}
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [WARN] - container runtime error
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: Caused by:
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]:     deadline has elapsed
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [WARN] - Failed to stop module: runtime operation error: stop module "edgeAgent"
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: Caused by:
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]:     0: container runtime error
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]:     1: deadline has elapsed
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - All modules stopped
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - watchdog finished
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Stopping management API...
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Stopping workload API...
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Management API stopped
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Workload Manager stopped
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Stopping listener for module endurance_cs_setup
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Stopping listener for module endurance_lte
Jan 18 09:21:21 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:21Z [INFO] - Stopping listener for module edgeAgent
Jan 18 09:21:21 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:21Z [INFO] - <-- POST /identities/device/reprovision?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
Jan 18 09:21:21 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:21Z [INFO] - Provisioning starting. Reason: Api
Jan 18 09:21:21 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:21Z [INFO] - Updated device info for ss-ped-dv1.
Jan 18 09:21:21 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:21Z [INFO] - Provisioning complete.
Jan 18 09:21:21 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:21Z [INFO] - Identity reconciliation started. Reason: Api
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:21 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:21Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:23Z [INFO] - Hub identity "DefenderIotMicroAgent" will be recreated
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "352"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "792"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "356"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "800"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:23 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:23Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:24 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:24Z [INFO] - Hub identity "DefenderIotMicroAgent" added
Jan 18 09:21:24 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:24Z [INFO] - Identity reconciliation complete.
Jan 18 09:21:24 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:24Z [INFO] - --> 204 {}
Jan 18 09:21:24 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:24Z [INFO] - Successfully reprovisioned
Jan 18 09:21:24 ss-ped-dv1 aziot-edged[31321]: 2024-01-18T09:21:24Z [INFO] - Device provisioning has changed. Restarting Edge daemon to get new provisioning info.
Jan 18 09:21:24 ss-ped-dv1 systemd[1]: aziot-edged.service: Main process exited, code=exited, status=154/n/a
Jan 18 09:21:24 ss-ped-dv1 systemd[1]: aziot-edged.service: Failed with result 'exit-code'.
Jan 18 09:21:29 ss-ped-dv1 systemd[1]: aziot-edged.service: Scheduled restart job, restart counter is at 475.
Jan 18 09:21:29 ss-ped-dv1 systemd[1]: Stopped Azure IoT Edge daemon.
Jan 18 09:21:29 ss-ped-dv1 systemd[1]: Started Azure IoT Edge daemon.
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Starting Azure IoT Edge Daemon
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Version - 1.4.27
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Obtaining Edge device provisioning data...
Jan 18 09:21:29 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:29Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Device is ss-ped-dv1 on iothub-endurance-prod-uksouth.azure-devices.net
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Initializing module runtime...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Using runtime network id azure-iot-edge
Jan 18 09:21:29 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Successfully initialized module runtime
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:29Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Jan 18 09:21:29 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Using existing Edge CA certificate
Jan 18 09:21:29 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:29Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Jan 18 09:21:29 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-28T11:28:57+00:00.
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping all modules...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping module endurance_cs_setup...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping module endurance_lte...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping module edgeAgent...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - All modules stopped
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Detecting if device information has changed...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Change to device information detected
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing all modules...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing module endurance_cs_setup...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing module endurance_lte...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing module edgeAgent...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removed all modules
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Updating cached device information
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Starting management API...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Starting workload API...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Starting watchdog with 60 second period...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Starting image garbage collection task...
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module endurance_cs_setup
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module endurance_lte
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module edgeAgent
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing listener for module edgeAgent
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module edgeAgent
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Watchdog checking Edge runtime status
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing listener for module endurance_cs_setup
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module endurance_cs_setup
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Removing listener for module endurance_lte
Jan 18 09:21:29 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:29Z [INFO] - Stopping listener for module endurance_lte
Jan 18 09:21:29 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:29Z [INFO] - <-- PUT /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "identityd.sock", "content-length": "40"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:29 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:29Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "736"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "748"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:30 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:30Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:30 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:30Z [INFO] - Creating and starting Edge runtime module edgeAgent...
Jan 18 09:21:30 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:30Z [INFO] - Pulling image via tag mcr.microsoft.com/azureiotedge-agent:1.4...
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Successfully pulled image mcr.microsoft.com/azureiotedge-agent:1.4
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Creating module edgeAgent...
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Creating image via tag mcr.microsoft.com/azureiotedge-agent:1.4...
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Created Edge runtime module edgeAgent
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Starting module edgeAgent...
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Starting new listener for module edgeAgent
Jan 18 09:21:31 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:31Z [INFO] - Starting workload API...
Jan 18 09:21:32 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:32Z [INFO] - Started Edge runtime module edgeAgent
Jan 18 09:21:33 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:33Z [INFO] - <-- GET /trust-bundle?api-version=2020-07-07 {"accept": "application/json", "host": "workload.sock:80", "connection": "close"}
Jan 18 09:21:33 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:33Z [INFO] - <-- GET /certificates/aziot-edged-trust-bundle?api-version=2020-09-01 {"host": "certd.sock"}
Jan 18 09:21:33 ss-ped-dv1 aziot-certd[1222]: 2024-01-18T09:21:33Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:33 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:33Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:34 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:34Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 18 09:21:34 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:34Z [INFO] - Querying system info...
Jan 18 09:21:34 ss-ped-dv1 aziot-edged[32358]: none
Jan 18 09:21:34 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:34Z [INFO] - Successfully queried system info
Jan 18 09:21:34 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:34Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:35 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:35Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 18 09:21:35 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:35Z [INFO] - Querying system info...
Jan 18 09:21:35 ss-ped-dv1 aziot-edged[32371]: none
Jan 18 09:21:35 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:35Z [INFO] - Successfully queried system info
Jan 18 09:21:35 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:35Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:36 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:36Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638346084343329829/sign?api-version=2018-06-28 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "193"}
Jan 18 09:21:36 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:36Z [INFO] - <-- GET /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
Jan 18 09:21:36 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:36Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:36 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:36Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:36 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:36Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:36 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:36Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - <-- GET /systeminfo?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - Querying system info...
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32395]: none
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - Successfully queried system info
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- GET /key/aziot_identityd_master_id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "736"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- POST /derivedkey?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "340"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- POST /derivedkey/export?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "748"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - <-- POST /sign?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "934"}
Jan 18 09:21:37 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - <-- GET /modules?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 18 09:21:37 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:37Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:38 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:38Z [INFO] - <-- POST /modules/%24edgeAgent/genid/638346084343329829/encrypt?api-version=2022-08-03 {"accept": "application/json", "host": "workload.sock:80", "connection": "close", "content-type": "application/json", "content-length": "7294"}
Jan 18 09:21:38 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:38Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "74"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:38Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - <-- POST /encrypt?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "7626"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- GET /identities/?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close"}
Jan 18 09:21:39 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:39Z [INFO] - <-- GET /identities/modules?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "398"}
Jan 18 09:21:39 ss-ped-dv1 aziot-keyd[587]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - --> 200 {"content-type": "application/json"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/endurance_lte?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "59"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/endurance_commander?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "59"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/endurance_can_broadcaster?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "59"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/endurance_watchdog?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "59"}
Jan 18 09:21:39 ss-ped-dv1 aziot-edged[32230]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/IoTEdgeMetricsCollector?api-version=2022-08-03 {"accept": "application/json", "host": "mgmt.sock:80", "connection": "close", "content-type": "application/json", "content-length": "59"}
Jan 18 09:21:39 ss-ped-dv1 aziot-identityd[611]: 2024-01-18T09:21:39Z [INFO] - <-- PUT /identities/modules/endurance_lte?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "identityd.sock", "content-length": "43"}

John Lian · Answer 2 · Fri Jan 19 2024 03:39:38 GMT+0800 (China Standard Time)

@vipeller (on-call) please help take a look

vipeller · Answer 3 · Fri Jan 19 2024 05:39:12 GMT+0800 (China Standard Time)

@guy-radford-sunswap I see a reprovisioning request in the log and a high restart counter, that indicates that the device keeps trying to reprovision. Isn't it possible that the device is disabled (e.g through the azure portal), that is one reason for continuous reprovisioning.

guy-radford-sunswap · Answer 4 · Fri Jan 19 2024 17:04:23 GMT+0800 (China Standard Time)

Neither of the affected devices have been disabled(not that I did or can see).
The first device worked fine for about 24 hours before entering this state and then stayed in this state for about 25 hours and now works again.
The second device went into this state immediately.

Both devices had the same new OS build written to them and then the IoT Device connection string was applied.

I compared versions of installed software between these and the development devices and that was fine.

After not being able to find anything in the Device configurations (I recreated for a test device) and ensuring the latest versions, I built a new OS build by hand and tested every step.

The hand build OS worked as expected, I have just finished the auto build and I will be rolling that out today.

The only thing that was possibly an issue for the two failing builds was:

the default Raspberry Pi Mirror was giving me an error, so I switched to a different mirror.
for the first time I used PiShirk, which shouldn't have caused an issue, but I have stopped using it until further testing.

Thank you

guy-radford-sunswap · Answer 5 · Fri Jan 19 2024 18:07:00 GMT+0800 (China Standard Time)

Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.GeneratedCode.SwaggerException: Response was null which was not expected.

Logs from edgeAgent with Debugging enabled.

root@ss-ped-dv1:~#  iotedge logs edgeAgent -f
2024-01-19 10:02:46  Starting Edge Agent
2024-01-19 10:02:46  Creating UID 13622 as edgeagentuser
2024-01-19 10:02:46  Changing ownership of storage folder: /tmp/edgeAgent/edgeAgent to 13622
2024-01-19 10:02:46  Creating backup folder: /tmp/edgeAgent_backup
2024-01-19 10:02:46  Changing ownership of management socket: /var/run/iotedge/mgmt.sock
2024-01-19 10:02:46  Completed necessary setup. Starting Edge Agent.
2024-01-19 10:02:46.823 +00:00 Edge Agent Main()
<6> 2024-01-19 10:02:47.420 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Initializing Edge Agent.
<6> 2024-01-19 10:02:47.864 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Version - 1.4.29.85541525 (f049017a5072f85aec250e945ebb2ca52e4af3f4)
<6> 2024-01-19 10:02:47.866 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] -
        █████╗ ███████╗██╗   ██╗██████╗ ███████╗
       ██╔══██╗╚══███╔╝██║   ██║██╔══██╗██╔════╝
       ███████║  ███╔╝ ██║   ██║██████╔╝█████╗
       ██╔══██║ ███╔╝  ██║   ██║██╔══██╗██╔══╝
       ██║  ██║███████╗╚██████╔╝██║  ██║███████╗
       ╚═╝  ╚═╝╚══════╝ ╚═════╝ ╚═╝  ╚═╝╚══════╝

 ██╗ ██████╗ ████████╗    ███████╗██████╗  ██████╗ ███████╗
 ██║██╔═══██╗╚══██╔══╝    ██╔════╝██╔══██╗██╔════╝ ██╔════╝
 ██║██║   ██║   ██║       █████╗  ██║  ██║██║  ███╗█████╗
 ██║██║   ██║   ██║       ██╔══╝  ██║  ██║██║   ██║██╔══╝
 ██║╚██████╔╝   ██║       ███████╗██████╔╝╚██████╔╝███████╗
 ╚═╝ ╚═════╝    ╚═╝       ╚══════╝╚═════╝  ╚═════╝ ╚══════╝

<6> 2024-01-19 10:02:47.880 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - ModuleUpdateMode: NonBlocking
<6> 2024-01-19 10:02:48.066 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<7> 2024-01-19 10:02:48.118 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to TrustBundleAsync
<7> 2024-01-19 10:02:48.185 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<7> 2024-01-19 10:02:48.231 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/workload.sock
<7> 2024-01-19 10:02:48.245 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://workload.sock/trust-bundle?api-version=2020-07-07
<7> 2024-01-19 10:02:48.348 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:48.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Received a valid Http response from unix:///var/run/iotedge/workload.sock for TrustBundleAsync
<6> 2024-01-19 10:02:48.476 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Installing certificates [CN=aziot-edge CA ss-ped-dv1:04/18/2024 09:29:31] to Root
<6> 2024-01-19 10:02:48.963 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Starting metrics listener on Host: *, Port: 9600, Suffix: metrics
<6> 2024-01-19 10:02:49.004 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Updating performance metrics every 05m:00s
<6> 2024-01-19 10:02:49.022 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Started operation Get system resources
<6> 2024-01-19 10:02:49.028 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Collecting metadata metrics
<7> 2024-01-19 10:02:49.049 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2024-01-19 10:02:49.076 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:49.078 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:49.079 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03
<7> 2024-01-19 10:02:49.119 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:49.215 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2024-01-19 10:02:49.256 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Set metadata metrics: 1.4.29.85541525 (f049017a5072f85aec250e945ebb2ca52e4af3f4), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"Linux","Architecture":"arm","Version":"1.4.27","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"24.0.7-1","KernelVersion":"#1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023","OperatingSystem":"raspbian","NumCpus":4,"TotalMemory":953647104,"Virtualized":"no"}, True
<6> 2024-01-19 10:02:49.313 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Metrics.DeploymentMetrics] - Started operation Checkpoint Availability
<6> 2024-01-19 10:02:49.323 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Started operation refresh twin config
<7> 2024-01-19 10:02:49.330 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Initialize twin refresh with interval '"01:00:00"'
<7> 2024-01-19 10:02:49.348 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Begin refreshing twin from upstream...
<7> 2024-01-19 10:02:49.356 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Getting module client to refresh the twin with retrying set to False
<6> 2024-01-19 10:02:49.391 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<7> 2024-01-19 10:02:49.525 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2024-01-19 10:02:49.525 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:49.527 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:49.529 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03
<7> 2024-01-19 10:02:49.593 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:49.595 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<6> 2024-01-19 10:02:50.248 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Modules.AgentModule] - Created persistent store at /tmp/edgeAgent/edgeAgent
<6> 2024-01-19 10:02:50.407 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Scrape
<6> 2024-01-19 10:02:50.407 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Diagnostics.MetricsScraper] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<7> 2024-01-19 10:02:50.423 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Waiting on shutdown handler to trigger
<7> 2024-01-19 10:02:51.537 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2024-01-19 10:02:51.538 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:51.539 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:51.540 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03
<7> 2024-01-19 10:02:51.569 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ConfigSources.TwinConfigSource] - TwinConfigSource Created
<7> 2024-01-19 10:02:51.583 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:51.584 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<7> 2024-01-19 10:02:51.627 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Edge agent config backup created here - /tmp/edgeAgent/edgeAgent/backup.json
<6> 2024-01-19 10:02:51.668 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadModuleLogs
<6> 2024-01-19 10:02:51.669 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler GetModuleLogs
<6> 2024-01-19 10:02:51.670 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler UploadSupportBundle
<6> 2024-01-19 10:02:51.672 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Core.Requests.RequestManager] - Registering request handler RestartModule
<7> 2024-01-19 10:02:51.770 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Edge agent created.
<7> 2024-01-19 10:02:51.781 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation
<7> 2024-01-19 10:02:51.832 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules
<7> 2024-01-19 10:02:51.854 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:51.855 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:51.855 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03
<7> 2024-01-19 10:02:51.862 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:51.878 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config...
<7> 2024-01-19 10:02:51.972 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules
<7> 2024-01-19 10:02:52.373 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected with reason Bad_Credential
<7> 2024-01-19 10:02:52.387 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to reprovision the device
<7> 2024-01-19 10:02:52.412 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:52.414 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:52.415 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/device/reprovision?api-version=2022-08-03
<7> 2024-01-19 10:02:52.455 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received NoContent
<6> 2024-01-19 10:02:52.484 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only...
<7> 2024-01-19 10:02:52.486 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info
<7> 2024-01-19 10:02:52.486 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:52.486 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock
<7> 2024-01-19 10:02:52.486 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03
<6> 2024-01-19 10:02:52.511 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Termination requested, initiating shutdown.
<6> 2024-01-19 10:02:52.512 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Waiting for cleanup to finish
<7> 2024-01-19 10:02:52.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK
<7> 2024-01-19 10:02:52.531 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info
<7> 2024-01-19 10:02:52.479 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Error when getting an Http response from unix:///var/run/iotedge/mgmt.sock for reprovision the device
HTTP Response:



Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.GeneratedCode.SwaggerException: Response was null which was not expected.

Status: 204
Response:

   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2022_08_03.GeneratedCode.EdgeletHttpClient.ReprovisionDeviceAsync(String api_version, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2022_08_03/generatedCode/EdgeletHttpClient.cs:line 1867
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.<>c__DisplayClass36_0.<<Execute>b__0>d.MoveNext() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 142
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 152
<7> 2024-01-19 10:02:53.344 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected_Retrying with reason Communication_Error
<4> 2024-01-19 10:02:56.897 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/edgeAgent/backup.json) instead
<7> 2024-01-19 10:02:56.914 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Making a Http call to unix:///var/run/iotedge/workload.sock to Decrypt
<7> 2024-01-19 10:02:56.961 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/workload.sock
<3> 2024-01-19 10:02:56.971 +00:00 [ERR] [Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Decrypt
System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)
   at System.Net.Sockets.SocketTaskExtensions.ConnectAsync(Socket socket, EndPoint remoteEP)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 30
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.DecryptAsync(String api_version, String name, String genid, DecryptRequest payload, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 352
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.DecryptAsync(String api_version, String name, String genid, DecryptRequest payload, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.DecryptAsync(String api_version, String name, String genid, DecryptRequest payload) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 308
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.<>c__DisplayClass6_0.<DecryptAsync>b__0() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 81
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.AsyncExecution`1.ExecuteAsyncImpl(Task ignore) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/AsyncExecution[T].cs:line 64
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.AsyncExecution`1.ExecuteAsync() in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/AsyncExecution[T].cs:line 43
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.RetryPolicy.ExecuteAsync[TResult](Func`1 taskFunc, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/RetryPolicy.cs:line 254
   at Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling.RetryPolicy.ExecuteAsync[TResult](Func`1 taskFunc) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/transientFaultHandling/RetryPolicy.cs:line 233
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.ExecuteWithRetry[T](Func`1 func, Action`1 onRetry, ITransientErrorDetectionStrategy transientErrorDetection) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 89
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.DecryptAsync(String initializationVector, String encryptedText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/WorkloadClient.cs:line 81
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.WorkloadClient.DecryptAsync(String initializationVector, String encryptedText)
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClient.DecryptAsync(String initializationVector, String encryptedText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClient.cs:line 30
   at Microsoft.Azure.Devices.Edge.Util.Edged.EncryptionProvider.DecryptAsync(String encryptedText) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/EncryptionProvider.cs:line 41
   at Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.DeploymentFileBackup.ReadFromBackupAsync() in /mnt/vss/_work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/configsources/DeploymentFileBackup.cs:line 57
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.Azure.Devices.Edge.Util.Edged.Version_2019_01_30.GeneratedCode.HttpWorkloadClient.DecryptAsync(String api_version, String name, String genid, DecryptRequest payload, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/version_2019_01_30/generatedCode/HttpWorkloadClient.cs:line 352
   at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task`1 task, TimeSpan timeout, Action action) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 135
   at Microsoft.Azure.Devices.Edge.Util.Edged.WorkloadClientVersioned.Execute[T](Func`1 func, String operation) in /mnt/vss/_work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/edged/WorkloadClientVersioned.cs:line 61
root@ss-ped-dv1:~#

guy-radford-sunswap · Answer 6 · Fri Jan 19 2024 18:52:01 GMT+0800 (China Standard Time)

This error, kept popping up:

[WRN] - Updating reported properties failed with error error(condition:amqp:unauthorized-access,description:Put token failed. status-code: 401, status-description: The specified SAS token has an invalid signature. It does not match either the primary or secondary key..) type Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException

I have been using the SAS Token for this device for many months without issue.... now it starts to fail?

Are there limits on Totken use?

I tried reconfiguring the device again with the original SAS Token and the error persisted.

I then regenerated the SAS Token, updated the config.toml and re-applied the config and now appears to be working.

Does any of this make sense?

John Lian · Answer 7 · Fri Jan 26 2024 03:13:09 GMT+0800 (China Standard Time)

@vipeller can you help with the SAS error?

@guy-radford-sunswap do you mean SAS key instead of token?

vipeller · Answer 8 · Fri Jan 26 2024 03:39:36 GMT+0800 (China Standard Time)

I think the sas key problem is related to the reprovisioning problem from the first log. This line:

2024-01-18T09:20:51Z [INFO] - Edge daemon will reprovision and restart

This causes two things:

sas keys will be regenerated for modules
the edge daemon will stop for a while

This would explain the token signature error and also the failure to call the management interface ("Shutting down because no response from unix:///var/run/iotedge/workload.sock")

However, it is not clear what causes reprovision. This usually happens with authentication problems, and I see one in the earlier logs:

<7> 2024-01-19 10:02:51.972 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules
<7> 2024-01-19 10:02:52.373 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected with reason Bad_Credential
<7> 2024-01-19 10:02:52.387 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to reprovision the device

This log line seems like Iot Hub does not think anymore that the edgeAgent connection is "right" and disconnects it. It would be nice to see the related IoT Hub cloud logs.

I see that the device name is "ss-ped-dv1", let me see if I can find some related cloud-side logs, why the authentication problem happens

guy-radford-sunswap · Answer 9 · Fri Jan 26 2024 16:37:15 GMT+0800 (China Standard Time)

@jlian

@guy-radford-sunswap do you mean SAS key instead of token?

Yes, sorry, SAS Key

@vipeller Thank you

vipeller · Answer 10 · Sat Jan 27 2024 07:46:26 GMT+0800 (China Standard Time)

@guy-radford-sunswap I've checked the azure service logs. I see that the problem started jan 17th ~11:43am (utc), then it went on continuously. The errors are "Unauthorized", with wrong token signature.

I see one suspicious thing: there are errors for a device with ip address 31.55.xx.xx and also 148.252.xx.xx. Both uses the same device id.

Isn't it possible that two device was configured with the same connection string?

In this case what may have happened:

one of the device got an error that made it reprovisioning (reprovisioning - it generates new module identities with new module keys). I am not sure what caused this 0th event, but for now let's say it happened somehow. Also let's say, that 148.252.xx.xx did this
Because of 0), now the modules have new keys. Let's say that the device from 31.55.xx.xx tries to connect, but it still knows about the old keys. This will cause an authorization error, and makes the device reprovision (=resets the module keys)
148.252.xx.xx is not happy that it has new keys and its modules are trying to connect using those keys. However, because of 1), the reprovisioning has reset these keys to new ones, and 148.252.xx.xx. modules will fail. This triggers a reprovisioning
31.55.xx.xx is also happy and does like 148.252.xx did in 2)

so at the end the two devices keep reseting eachothers keys.

When you regenerated the device keys, one of the device was not able to reset anymore, and the other managed to reprovision once, then worked normally.

I don't see every details here (like 0), but do you think that something like that could happen? Does those 2 ip addresses tell you something?

guy-radford-sunswap · Answer 11 · Sun Jan 28 2024 01:15:03 GMT+0800 (China Standard Time)

@vipeller
Judging from what you have described, I copied the wrong SAS Key to the wrong device during provisioning. Then, I re-provisioned the second device with the correct key, which I had accidentally used on the first device, this kicked off all the issues.

If I had realised the first device was the issue then it would have all been an easy fix.

Thank you, I appreciate your help in getting to the bottom of this.

bishal41 · Answer 12 · Sat Feb 17 2024 06:33:07 GMT+0800 (China Standard Time)

@guy-radford-sunswap great to hear the issue is resolved for you. Closing this issue. Thanks!

edgeAgent keep restarting after [ERR] - Shutting down because no response from unix:///var/run/iotedge/workload.sock for Encrypt

Expected Behavior

Current Behavior

Steps to Reproduce

Context (Environment)

Output of iotedge check

Device Information

Runtime Versions

Logs

Additional Information

Output of `iotedge check`