Healthcheck not picking up 'error sem_timedwait' as error
SingingDwarf opened this issue · comments
So I noticed that my reader had failed overnight and was not collecting flight information.
Checking Portainer, I saw that some of the other containers were showing unhealthy, as they could not connect to readsd, however the readsb container itself was showing as healthy. Checking the readsb logs, the following error was repeated, but it had not triggered the unhealthy status:
[readsbrrd] 2023/03/23 09:16:50 error sem_timedwait: Connection timed out, semcnt: 0
Thanks for reporting this, and sorry you had a problem.
The log entry you've posted is for "readsbrrd". This utility collects stats from readsb for graphing and is likely a symptom of something else wrong.
Are you able to post more container logs from when the problem occurred?
Thanks.
Apologies, but there doesn't appear to be anything more useful really within the log.
This is the extract from the log around the time that graphing reports no new data around 22:30:
(NB The spurious entry at 2023/03/20 21:50:49 is just a straight copy/paste of the log!
[collectd] 2023/03/22 22:30:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:31:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:32:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:33:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:34:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:35:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:36:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:37:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:38:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/20 21:50:49 could not lock RRD
[readsbrrd] 2023/03/22 22:36:19 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:37:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:39:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:39:19 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:40:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:40:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:41:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:42:19 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:42:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:43:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:44:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:43:49 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:45:19 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:45:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:46:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:46:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:47:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:48:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:48:19 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:49:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:49:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:50:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:51:19 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:51:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:52:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:53:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:52:49 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:54:19 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:54:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:55:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:55:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:56:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:57:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 22:57:19 error sem_timedwait: Connection timed out, semcnt: 0
[readsbrrd] 2023/03/22 22:58:49 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 22:58:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[collectd] 2023/03/22 22:59:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
[readsbrrd] 2023/03/22 23:00:19 error sem_timedwait: Connection timed out, semcnt: 0
[collectd] 2023/03/22 23:00:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory.
You're right. Unfortunately there's not enough in there to diagnose the problem.
To fix the [collectd] 2023/03/22 22:58:50 [error] table plugin: Failed to open file "/sys/class/thermal/thermal_zone0/temp": No such file or directory. error, you may wish to implement https://github.com/sdr-enthusiasts/docker-radarbox/blob/main/version_0.4.3_workarounds.md#workaround-for-temperature-sensor. Note that this instruction was written for the radarbox container. Let me know if you need assistance modifying for the readsb container.
No worries - I'll close the issue for now and keep an eye out for similar in the future.
Thanks for the advice re the temp sensor issue, just in the process of deploying that now
No problems! Good luck.