Can't start hassio_observer: 500 Server Error

Question

Can't start hassio_observer: 500 Server Error

otakon87 opened this issue 4 years ago · comments

Hi, I use hassio on proxmox. Today I found it non responsive, restarted the VM and now I have this error showing up over and over again. What's happening? Thank you.

Supervisor
Version
2020.11.0
Operating System
HassOS 4.17
Home Assistant 0.118.4

20-12-02 14:08:40 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
20-12-02 14:08:40 INFO (SyncWorker_1) [supervisor.docker.interface] Cleaning hassio_observer application
20-12-02 14:08:43 ERROR (SyncWorker_1) [supervisor.docker] Can't start hassio_observer: 500 Server Error: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (f16a6062331efa497a8a628c3ed567aa13dd7bdb55a561315e9dab8072186307): Bind for 0.0.0.0:4357 failed: port is already allocated")
20-12-02 14:08:43 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
20-12-02 14:08:43 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!
20-12-02 14:09:43 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Docker found a problem with observer plugin!

Egor Ramanko · Answer 1 · Fri Dec 11 2020 05:25:59 GMT+0800 (China Standard Time)

Hi, I get the same error on my Raspberry Pi 3:

20-12-10 21:18:12 ERROR (SyncWorker_0) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.40/containers/d0e2f10cf7314c0226c6dc43475cc4156e13ca30a2fba7187cf2ce7b314572a3/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (19ba848aa792690771bcfe84270cf74399e1fd9754e9dbf34c4005e459773ff7): Bind for 0.0.0.0:4357 failed: port is already allocated")

Gleras · Answer 2 · Wed Dec 23 2020 02:12:05 GMT+0800 (China Standard Time)

Hi. same error on pi 3

Version
2020.12.7

Operating System
Raspbian GNU/Linux 10 (buster)
Docker version
19.03.13

20-12-22 17:54:48 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Docker found a problem with observer plugin!
20-12-22 17:54:48 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
20-12-22 17:54:48 INFO (SyncWorker_3) [supervisor.docker.interface] Cleaning hassio_observer application
20-12-22 17:54:49 WARNING (SyncWorker_3) [supervisor.docker] Can't attach hassio_observer to hassio-network!
20-12-22 17:54:49 ERROR (SyncWorker_3) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.40/containers/0781b6f53673d556df072ed32a2fb1f46f7788051dcbaddbb1a26f183119c61e/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (ff241f2c5ba3b5b83448a704713846ed11953213581a779c58d424cc64d75c55): Bind for 0.0.0.0:4357 failed: port is already allocated")
20-12-22 17:54:49 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
20-12-22 17:54:49 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!

majdzik84 · Answer 3 · Wed Dec 23 2020 16:45:36 GMT+0800 (China Standard Time)

Helo. same error on pi 3b+

Gleras · Answer 4 · Wed Dec 23 2020 16:52:34 GMT+0800 (China Standard Time)

fixed by reboot raspbian. will see

majdzik84 · Answer 5 · Wed Dec 23 2020 16:56:13 GMT+0800 (China Standard Time)

fix is temporary. The error returns again after a certain period.

Egor Ramanko · Answer 6 · Wed Dec 23 2020 17:29:02 GMT+0800 (China Standard Time)

I confirm, rebooting helps only for some time. The issue appears again.

Egor Ramanko · Answer 7 · Wed Dec 23 2020 17:30:02 GMT+0800 (China Standard Time)

@pvizeli @balloob hello, guys, are you aware of this issue?

frkos · Answer 8 · Thu Dec 24 2020 18:16:47 GMT+0800 (China Standard Time)

I have the same issue on my rpi 3b+ and latest version of HA

AcidSleeper · Answer 9 · Sat Dec 26 2020 19:59:53 GMT+0800 (China Standard Time)

Same here. Rpi3b+ and HA 0.118.5. Supervisor 2020.12.7 and OS 5.8

Dr John Tunnicliffe · Answer 10 · Mon Jan 11 2021 02:08:23 GMT+0800 (China Standard Time)

Update: 12-Jan: After restarting my Raspberry Pi, the problem went away. Not sure why the observer did not start previously

Same issue. HA Supervised installation on recently rebuilt Raspberry Pi4 with Debian 10 OS.

Supervisor version 2020.12.7
Operating System: Debian GNU/Linux 10 (buster)
Docker version: 20.10.1

Home Assistant Version	2021.1.1
Installation Type	Home Assistant Supervised
Development	false
Supervisor	true
Docker	true
Virtual Environment	false
Python Version	3.8.7
Operating System Family	Linux
Operating System Version	5.9.0-0.bpo.2-arm64
CPU Architecture	aarch64
Timezone	Europe/London

Full supervisor log after start up. Last lines repeat many times.

21-01-10 17:51:30 INFO (MainThread) [supervisor.host.services] Updating service information
21-01-10 17:51:31 INFO (MainThread) [supervisor.host.network] Updating local network information
21-01-10 17:51:34 INFO (MainThread) [supervisor.host] Host information reload completed
21-01-10 17:51:35 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_appdaemon
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
21-01-10 17:51:35 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
21-01-10 17:52:30 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Docker found a problem with observer plugin!
21-01-10 17:52:30 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
21-01-10 17:52:30 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_observer application
21-01-10 17:52:31 ERROR (SyncWorker_0) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.41/containers/2e8847f775b0b0561ad6f182387531a18530c26cb22f5f978390256b971f8b22/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (dfc0d2e09a765a09a892167a01a00eb70e6e9022fb547426c174db661e79c3c6): Bind for 0.0.0.0:4357 failed: port is already allocated")
21-01-10 17:52:31 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
21-01-10 17:52:31 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!
21-01-10 17:53:31 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Docker found a problem with observer plugin!
21-01-10 17:53:31 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
21-01-10 17:53:31 INFO (SyncWorker_4) [supervisor.docker.interface] Cleaning hassio_observer application
21-01-10 17:53:31 ERROR (SyncWorker_4) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.41/containers/1a508611405460412f63fb6a670087d46469d52f9bbe8db13188633b7c541d55/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (2cdac0fa0f8014468e472609225792fe86fd57a1997985aa1a31ac234e16162c): Bind for 0.0.0.0:4357 failed: port is already allocated")
21-01-10 17:53:31 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
21-01-10 17:53:31 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!
21-01-10 17:54:31 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Docker found a problem with observer plugin!
21-01-10 17:54:31 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
21-01-10 17:54:31 INFO (SyncWorker_6) [supervisor.docker.interface] Cleaning hassio_observer application
21-01-10 17:54:31 ERROR (SyncWorker_6) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.41/containers/f5d6c3f32ee244a2c5effaf5db6927c55d2b2aa3ec6aa9dc4d47d33b7222ea25/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (a8d601ba5b7d4030acc85d9e4e298bc43ba738e7b35e3f4eb34d8de965c174a7): Bind for 0.0.0.0:4357 failed: port is already allocated")
21-01-10 17:54:31 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
21-01-10 17:54:31 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!

QbaF · Answer 11 · Wed Jan 13 2021 01:06:11 GMT+0800 (China Standard Time)

Same here onRPi4 @hassos

backba · Answer 12 · Wed Jan 13 2021 01:43:19 GMT+0800 (China Standard Time)

I have had the same issue for months, but yesterday I read somewhere that this is caused by lack of memory in my raspberryPi 3 (1gb ram). And that integrations are consuming to much memory. So I started by removing HACS and haven’t got an issue since. (Just took about a few hours after restart for the error to come back before). If it would show up again I will remove next integration and so on.

Joakim Sørensen · Answer 13 · Wed Jan 13 2021 18:31:18 GMT+0800 (China Standard Time)

Memory usage does not cause port conflicts.
Other applications that are running might.
If not, run ha su repair

backba · Answer 14 · Wed Jan 13 2021 19:41:43 GMT+0800 (China Standard Time)

I agree, but in my case it really have solved my issue. I’m just speculating, but could it be that Hassio observer could not be started because there already is one instance running (but hanged) and blocking the port?

Joakim Sørensen · Answer 15 · Wed Jan 13 2021 19:43:07 GMT+0800 (China Standard Time)

yes, that would definitely do it, the repair command will clean that up.

backba · Answer 16 · Wed Jan 13 2021 19:50:52 GMT+0800 (China Standard Time)

Ah! When I tested the repair command it solved it first, but the issue came back a few hours later anyway.. since deleting HACS I haven’t seen the error (3 days ago). Maybe it could be me that made a bad installation of HACS integration somehow.

Aladin · Answer 17 · Sun Jan 17 2021 21:32:58 GMT+0800 (China Standard Time)

I have same problem and saw that port 4357 was used in tcp6 services usage but if
I suppress ipv6 in my system.conf and reboot, I still have the tcp6 using that port

tcp6 0 0 [::]:4357 [::]:*

How to fix it ?

summongod · Answer 18 · Wed Jan 27 2021 20:46:14 GMT+0800 (China Standard Time)

I'm seeing the same problem and get the message below repeating continually in my logs, I have run the repair command as mentioned earlier in this thread and the problem persists. I'm suspicious that this is causing me other issues on my system as automations are failing to trigger fairly frequently.
Is it possible to disable the Observer plugin?

21-01-27 12:30:03 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
21-01-27 12:30:03 INFO (SyncWorker_5) [supervisor.docker.interface] Cleaning hassio_observer application
21-01-27 12:30:07 WARNING (SyncWorker_5) [supervisor.docker] Can't attach hassio_observer to hassio-network!
21-01-27 12:30:16 ERROR (SyncWorker_5) [supervisor.docker] Can't start hassio_observer: 500 Server Error for http+docker://localhost/v1.40/containers/17af5a0d7c8e0ad673df0eb8f35580a29c9eb91872a44d4456c32c463983fcd0/start: Internal Server Error ("driver failed programming external connectivity on endpoint hassio_observer (7612809336ca3eb4c8a37f4eab0227a0e0191caa5fa972f2820eedeebd12d468): Bind for 0.0.0.0:4357 failed: port is already allocated")
21-01-27 12:30:16 ERROR (MainThread) [supervisor.plugins.observer] Can't start observer plugin
21-01-27 12:30:16 ERROR (MainThread) [supervisor.misc.tasks] Observer watchdog reanimation failed!

summongod · Answer 19 · Fri Jan 29 2021 01:10:29 GMT+0800 (China Standard Time)

I fixed this on my RPi with Hassos by removing IPv6 on the host and rebooting.

Brian Moore · Answer 20 · Wed Feb 03 2021 06:07:10 GMT+0800 (China Standard Time)

Similar issue for me; only temporary fix in order to even investigate is hard reboot of my RPi 3b+... Seems to reboot fine but eventually poop out.

System Health

version	2021.1.5
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
virtualenv	false
python_version	3.8.7
os_name	Linux
os_version	5.4.79-v8
arch	aarch64
timezone	America/Los_Angeles

Home Assistant Community Store

GitHub API	ok
Github API Calls Remaining	4921
Installed Version	1.10.1
Stage	running
Available Repositories	711
Installed Repositories	4

Hass.io

host_os	Home Assistant OS 5.10
update_channel	stable
supervisor_version	2021.01.7
docker_version	19.03.13
disk_total	13.9 GB
disk_used	10.3 GB
healthy	true
supported	true
board	rpi3-64
supervisor_api	ok
version_api	ok
installed_addons	Grafana (6.0.0), Samba share (9.3.0), Terminal & SSH (8.10.0), Google Assistant SDK (2.5.0), Plex Media Server (2.4.1), Duck DNS (1.12.4), Visual Studio Code (2.9.1), File editor (5.2.0), Mosquitto broker (5.1), OpenZWave (0.9.0), NGINX Home Assistant SSL proxy (3.0.1), Let's Encrypt (4.11.0), Node-RED (7.2.8), ESPHome (1.15.3), Assistant Relay (0.7.4), Z-Wave Network Viewer (0.4)

Lovelace

dashboards	1
mode	storage
views	5
resources	1

summongod · Answer 21 · Wed Feb 03 2021 06:34:13 GMT+0800 (China Standard Time)

Check to see if a service is using port 4357, if it is kill the service. If it IPv6 again that means you didn't remove IPv6 support from the host.

Leonardo Properzi · Answer 22 · Fri Apr 09 2021 20:40:35 GMT+0800 (China Standard Time)

@summongod How can I disable ipv6? I'm on hassio rpi0w, editing /etc/sysctl.conf would do?

TasmanDevil · Answer 23 · Thu Aug 19 2021 04:47:26 GMT+0800 (China Standard Time)

I assume that the setting is in Supervisor>System>IP address (CHANGE)

Anyway I have this issue too :/

@summongod How can I disable ipv6? I'm on hassio rpi0w, editing /etc/sysctl.conf would do?

Joakim Sørensen · Answer 24 · Wed Sep 29 2021 22:39:43 GMT+0800 (China Standard Time)

If anyone still sees this, open an issue in the supervisor repo.