HA install variable results
sddgit opened this issue · comments
I’v been running your excellent script for some time, and have just tried it running under HA. The results have been mixed. Sometimes it works, sometimes it doesn’t. I see no errors in the HA log when it doesn’t work. The log looks OK, except it has no message indicating a successful connection, even when it works:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Sometimes I get a string of timeout errors after some time, but I can ping the Envoy fine from HA. Sometimes it will take quite a few minutes before it starts. All very variable really!
Can you think of any troubleshooting I can try? I can pretty easily get it to the point of not working by simply stopping and starting it a few times.
Interesting, I've never had a problem, it starts and runs reliably without any issues. It even gracefully stops/restarts after the HA OS is updated as I did this morning.
What device are you running HA on? Mine is running as a virtual machine in a macbook pro thats been repurposed and is running Linux Mint. I assume you previously ran the script outside of HA on the native host itself? I have noticed if i start the addon and then go immediately to the log, it seems to miss a lot of output - if i start it and wait about 30 second then look at the logs it will always display the full message. I doubt it's my code doing this as it seems to be a problem with the addon store panel.
Have you tried using the windows/mac/linux app called Mqtt explorer? http://mqtt-explorer.com/ - point this at your broker and you'll see the raw real-time output from the script which might help your fault finding.
Further fault-finding you could try - copy all the files to a new directory you create in addons, it will then show-up as a new Local addon section.
It's certainly weird. If you manually stop the add-on, then start it again, how long does it take to kick in? Mine sometimes starts quickly, but often doesn't seem to kick in at all, and sometimes after a few minutes. MQTT Explorer shows no activity at all on the /envoy/json topic when it doesn't start properly. Even after waiting a long while, the log doesn't show the "connected" message.
I'm running on an HA Blue box, Hassos 7.6, Core 2022.4.1 (all the latest). When I stop and restart the add-on, this appears in the main HA log, which I suppose is correct:
22-04-09 21:42:10 INFO (SyncWorker_0) [supervisor.docker.interface] Stopping addon_13e68335_envoy_to_mqtt_json application
22-04-09 21:42:11 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning addon_13e68335_envoy_to_mqtt_json application
22-04-09 21:42:45 INFO (SyncWorker_2) [supervisor.docker.addon] Starting Docker add-on 13e68335/aarch64-addon-envoy_to_mqtt_json with version 1.0.3
22-04-09 21:42:46 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'mqtt'
22-04-09 21:42:46 INFO (MainThread) [supervisor.auth] Successful login for 'mqtt'
Playing with stuff in Addons is a bit beyond me at present. How would I even get to that directory, and where exactly would the add-on show up?
Thanks for the follow-up!
Edit: I previously ran the script on a separate MacOS machine. When it starts in that environment, the sensors in HA update immediately.
Don't worry about adding it as a local addon, it was only if you wanted to do some of your own in depth testing. Mine starts within a few seconds and the HA log looks normal. Maybe increase the HA log detail to see if any other messages appear - however I'm not sure how to do that with the Docker addons -
This may be asking too much, but is there any chance you could add some more diagnostics to your own logging at critical steps (with time stamps)? Like “logging on to Envoy” “successful logon”, “received first stream”, “Logged on to broker” etc…
@sddgit - there's already diagnostics within the code that will display errors in the log if there's a problem - however I just made a change to force the script to wait 10 seconds from the initial connection to the mqtt broker to when it starts streaming the data to it - can you give that a try v1.04?
I can see one change with V1.04. If I keep refreshing the log I can see the “Connected to” message now. But I still get very long wait times sometimes before the topic is published. My thought on the log messages was to try to determine if something was taking a long time to accomplish, like logging into the Envoy or the broker. I do sometimes see a bunch of Envoy timeout errors before the streaming starts to work, which doesn’t happen when using the old script. All very puzzling!
@sddgit - my log shows this - can you copy/paste yours? Also, as a test, in configuration please change the ip address of the mqtt host to a wrong ip then restart it - you should see an error after refreshing the log. Then put back the correct mqtt ip and then change the ip of the envoy to a wrong ip - you should also see many errors when refreshing the logs. Maybe copy/paste those errors here so I can check.
s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
10/04/2022 15:46:54 Connected to 192.168.1.74:1883
10/04/2022 15:46:54 Connected to 192.168.1.74:1883
/envoy/json
I just stopped the add-on and started it again. It took about a minute for my power gauges to start showing any updates. During this time, I can go to the Mosquitto Broker integration, then Configure, then start listening for /envoy/json. There are no updates there either for the same length of time. Then it kicks in. After this, the log is:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
10/04/2022 19:17:45 Connected to 192.168.0.240:1883
I never see the second "Connected to" message or the /envoy/json lines, even when things eventually work.
(Incidentally, if I go to Add-ons, Backups & Supervisor, System tab, then look at the Multicast or DNS logs, some of your log messages seem to be there too)
Log with bad mqtt IP address:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
Traceback (most recent call last):
File "//envoy_to_mqtt_json.py", line 131, in <module>
client.connect(MQTT_HOST,int(MQTT_PORT), 30)
File "/usr/lib/python3.9/site-packages/paho/mqtt/client.py", line 941, in connect
return self.reconnect()
File "/usr/lib/python3.9/site-packages/paho/mqtt/client.py", line 1075, in reconnect
sock = self._create_socket_connection()
File "/usr/lib/python3.9/site-packages/paho/mqtt/client.py", line 3546, in _create_socket_connection
return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
File "/usr/lib/python3.9/socket.py", line 843, in create_connection
raise err
File "/usr/lib/python3.9/socket.py", line 831, in create_connection
sock.connect(sa)
OSError: [Errno 113] Host is unreachable
[cmd] /run.sh exited 1
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
Log with bad Envoy IP address:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
10/04/2022 19:28:55 Connected to 192.168.0.240:1883
10/04/2022 19:28:55 Connected to 192.168.0.240:1883
/envoy/json
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb7003400>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8179e80>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb817c040>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb81809a0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb81811f0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8188b80>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8180700>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb817be50>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8179df0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8188a30>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb70032b0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb7003d00>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8179f40>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb817bd30>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb81805b0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb81811c0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8188f70>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb7003190>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb81888b0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8181070>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8180550>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8180ac0>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb817ba90>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb8174f40>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
10/04/2022 19:28:55 Exception fetching stream data: HTTPConnectionPool(host='192.168.0.230', port=80): Max retries exceeded with url: /stream/meter (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffffb6dbe550>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
It took a while and a lot of refreshes until the errors popped up though.
Thanks for that, the errors are correct but you should be seeing the second "Connected to" message and the /envoy/json message.
So you're saying if you put in wrong IP details it takes a while for the logs to display? There should be a delay of only 10 seconds from when you start till you see the logs - if it takes longer then there's something funny happening locally within your HA Blue box. Are there any configuration options in Blue box for connections or networking?
Yes, with the wrong IP address, the logs take quite a while to display. But notice that the time stamp on the Exception messages is the same as the “Connected” messages, so it looks like it’s a logging delay rather than some sort of network issue. Although whatever is causing that delay might be responsible for the overall delay when it works I guess. The HA Blue is just like a Rpi - runs a complete flashed image, so there’s nothing to configure other than what’s in HA.
I have an HA that runs in a VirtualBox VM. When I get some time, I’ll crank that up, update it, and test the add-on there. Are you running 2022.4.1?
Yes I'm on 2022.4.1 - yes if you try running on a different machine it might help pinpoint where the issues are
OK. I got my VM up to date, and the add-on worked fine. It started publishing the topic after the 10 second wait after starting the add-on. The log file also showed the 2 connected message and /envoy/json. However, after a few tests, there were instances where the last one or two lines were missing.
So I went back to my production HA. I have done lots of HA restarts, but this time I did a host reboot (which I think I've tried before). It seems to be working now - starts publishing after 10 seconds. However, on this instance the log file is always missing the last 2 lines, unless something else happens, like Envoy timeouts, in which case things get flushed out. Not sure where the fault with this lies!
Thanks for bearing with me. I'll keep testing over the next couple of days - hopefully things will remain stable.
Hi @sddgit, I have noticed Home Assistant sometimes struggles with hostnames when I reboot my router.
Please see https://community.home-assistant.io/t/local-dns/178108/119?u=del13r for explanation regarding that issue.
Have you tried replacing the MQTT_HOST and envoy_host with an IP address instead?
Yes - I had one incident where HA wasn't able to resolve the hostname envoy.local so I've now using the IP address. Since this isn't related to this integration I'll close off this issue, thanks for your help.
@vk2him @del13r Thanks for following up. I have only ever used IP addresses. The add-on seems to be working fine now. The last few times I’ve started it, there is just the 10 second delay before it starts reporting. Still always missing the last couple of log lines, but that’s no big deal. Great to have this working inside HA!