Nut client disconnecting and re-connecting every minute (after WLAN reconnects)

Question

Nut client disconnecting and re-connecting every minute (after WLAN reconnects)

valeriansaliou opened this issue 3 years ago · comments

Hello,

First of all, thanks for your Homebridge plugin.
I'm running Homebridge latest on a RPi Zero W (connected over Wi-Fi) + homebridge-nut fetching UPSd data every 15 seconds from another LAN IP, which is wired over ethernet.

Once every two weeks, my Wi-Fi AP restarts (due to an AP firmware issue out of my control that wont be fixed), and thus all Wi-Fi devices lose connectivity for ~30s or so, until they reconnect. The main LAN router does not restart, so all wired connections are still up.

When the Wi-Fi reconnection happens, my Homebridge starts reporting every minute the following logs:

Again, and again, and again. If I restart the Homebridge service from the Homebridge admin UI, then the homebridge-nut plugin starts up and stops flooding the logs w/ reconnection logs every minute... until the next Wi-Fi reconnection happens a few days/weeks later, and this resumes.

This is my homebridge-nut config:

{
    "name": "Nut",
    "host": "10.0.1.5",
    "port": 3493,
    "low_batt_threshold": 40,
    "poll_interval": 15,
    "connect_interval": 5,
    "command_interval": 2,
    "platform": "Nut"
}

Would there be any issue in this plugin, or in the underlying nut client library used?

Note that if I put my UPS power down, homebridge-nut correctly reports it as OB even if it is actively flooding the logs, so this doesn't break the functionality of the plugin, it only floods the logs in a weird way.

Thanks!

Valerian.

vectronic · Answer 1 · Fri Feb 11 2022 01:11:37 GMT+0800 (China Standard Time)

Hmm interesting, give me some time and will check it out on the weekend...

vectronic · Answer 2 · Mon Feb 14 2022 02:11:02 GMT+0800 (China Standard Time)

Hi there,

Looking into this I can't really explain it. The actual nut client is pretty simple and is really just a wrapper around a NodeJS socket instance.

I also can't see where the interval of approx 55s between close events comes from...

One thought, are you looking at the home bridge error logs as well? These tend to be logged out separately and maybe you are getting some error there which could explain things.

Regardless, I have done a change (try v2.4.2) which completely discards the nut client and creates a new one when it connects. This may at least discard the old client when the issue occurs and the new one may no longer have the issue. Let me know if it helps...

Valerian Saliou · Answer 3 · Mon Feb 14 2022 02:14:52 GMT+0800 (China Standard Time)

@vectronic thanks, I'm updating the plugin right now. I'll keep you posted if this happens again.

Regarding the Homebridge error logs in /var/lib/homebridge/homebridge.log, they show exactly the same nut-related log lines as the Homebridge Web UI logs.

Valerian Saliou · Answer 4 · Sun Feb 27 2022 04:54:44 GMT+0800 (China Standard Time)

Okay, just had a Wi-Fi disconnection/reconnection and now this is happening again on the update you provided.

Logs:

[26/02/2022, 21:52:13] [Nut] nut client connected, reported devices: eaton=Eaton 3S 550
[26/02/2022, 21:53:12] [Nut] nutClose(hadError: undefined)
[26/02/2022, 21:53:12] [Nut] nutClose(hadError: undefined)
[26/02/2022, 21:53:16] [Nut] creating nut client for 10.0.1.5:3493
[26/02/2022, 21:53:16] [Nut] starting nut client for 10.0.1.5:3493
[26/02/2022, 21:53:18] [Nut] nut client connected, reported devices: eaton=Eaton 3S 550
[26/02/2022, 21:54:17] [Nut] nutClose(hadError: undefined)
[26/02/2022, 21:54:21] [Nut] creating nut client for 10.0.1.5:3493
[26/02/2022, 21:54:21] [Nut] starting nut client for 10.0.1.5:3493
[26/02/2022, 21:54:23] [Nut] nut client connected, reported devices: eaton=Eaton 3S 550

dragosde · Answer 5 · Thu Mar 03 2022 04:22:10 GMT+0800 (China Standard Time)

Following.

I have the same problem:

[02/03/2022, 15:17:56] [Nut] nutClose(hadError: undefined)
[02/03/2022, 15:17:58] [Nut] creating nut client for 192.168.2.96:3493
[02/03/2022, 15:17:58] [Nut] starting nut client for 192.168.2.96:3493
[02/03/2022, 15:18:01] [Nut] nut client connected, reported devices: APC1500=Description unavailable
[02/03/2022, 15:18:56] [Nut] nutClose(hadError: undefined)
[02/03/2022, 15:18:58] [Nut] creating nut client for 192.168.2.96:3493
[02/03/2022, 15:18:58] [Nut] starting nut client for 192.168.2.96:3493
[02/03/2022, 15:19:01] [Nut] nut client connected, reported devices: APC1500=Description unavailable
[02/03/2022, 15:19:56] [Nut] nutClose(hadError: undefined)
[02/03/2022, 15:19:58] [Nut] creating nut client for 192.168.2.96:3493
[02/03/2022, 15:19:58] [Nut] starting nut client for 192.168.2.96:3493
[02/03/2022, 15:20:01] [Nut] nut client connected, reported devices: APC1500=Description unavailable

vectronic · Answer 6 · Thu Mar 03 2022 05:18:27 GMT+0800 (China Standard Time)

Ok - thanks for the report.

I will look further when I get some time in the next week...

vectronic · Answer 7 · Fri Mar 04 2022 08:12:19 GMT+0800 (China Standard Time)

I've published a new version. If I understand my own code, there may have been a scenario where continually connection attempts occur. No promises whether this fixes the issue, but it's worth trying...

dragosde · Answer 8 · Fri Mar 04 2022 14:21:56 GMT+0800 (China Standard Time)

I guess you understand your code! I updated the plugin and all seems well. It solved it for me. Thanks!

Valerian Saliou · Answer 9 · Fri Mar 04 2022 16:21:57 GMT+0800 (China Standard Time)

Thanks, updated and will keep you posted about any further issue / or fixed issue.

dragosde · Answer 10 · Sun Mar 06 2022 02:30:52 GMT+0800 (China Standard Time)

It may be me, or rather my setup, but I still get the re-connections:

[05/03/2022, 13:16:49] [Nut] nutClose()
[05/03/2022, 13:16:51] [Nut] creating nut client for 192.168.2.96:3493
[05/03/2022, 13:16:51] [Nut] starting nut client for 192.168.2.96:3493
[05/03/2022, 13:16:54] [Nut] nut client connected, reported devices: APC1500=Description unavailable
[05/03/2022, 13:17:54] [Nut] nutClose()
[05/03/2022, 13:17:56] [Nut] creating nut client for 192.168.2.96:3493
[05/03/2022, 13:17:56] [Nut] starting nut client for 192.168.2.96:3493
[05/03/2022, 13:17:59] [Nut] nut client connected, reported devices: APC1500=Description unavailable
[05/03/2022, 13:18:59] [Nut] nutClose()
[05/03/2022, 13:19:01] [Nut] creating nut client for 192.168.2.96:3493
[05/03/2022, 13:19:01] [Nut] starting nut client for 192.168.2.96:3493
[05/03/2022, 13:19:04] [Nut] nut client connected, reported devices: APC1500=Description unavailable
[05/03/2022, 13:20:04] [Nut] nutClose()

I will have a look in my setup and see what I can improve and do some digging.

vectronic · Answer 11 · Mon Mar 14 2022 06:06:51 GMT+0800 (China Standard Time)

It seems the client socket connection to the nut server works ok, then the socket connection is closed 60 seconds later.

And I think I have found the culprit on the UPSD server side here:

https://github.com/networkupstools/nut/blob/master/server/upsd.c#L1031

If you could tell me your configured polling interval (which I suspect is greater than 60 seconds) then this would confirm the scenario. You could try changing your polling interval to 30 seconds and see if the problem disappears.

If this works, I might need to create an internal ping every 30 seconds to prevent the server from kicking us off

OR

Just handle being kicked off - does being disconnected every minute cause issues for the HomeKit accessory? Or does it just look odd in the logs?

dragosde · Answer 12 · Mon Mar 14 2022 08:58:57 GMT+0800 (China Standard Time)

My polling interval is exactly 60 seconds. I believe this is the default when one installs the plugin.
And believe you are right, it seems to be something on the server side. When I restart the the server the disconnection stops.
I kinda suspected that since I have restarted the server and for a while I saw no mentions in the homebridge log. This is what I hinted at when I wrote earlier that I need to check my setup. I thought I was doing something wrong.
It is worth mentioning that I have the upsd server on another pi, not on the one where my homebridge runs.
And indeed, I don't see any adverse effects other than the pollution of the homebridge logs.
If I get the disconnection logs again I will play with the polling interval. It is weird that I only get it sporadically, not constantly.

vectronic · Answer 13 · Mon Mar 14 2022 17:15:46 GMT+0800 (China Standard Time)

Thanks for the info. I am thinking I might just reduce the plug-in logging to debug instead of info as everything is behaving as it should. Alternatively it might be neater if I disconnect each polling cycle rather than leave the server to close the connection. Will have a think…

Valerian Saliou · Answer 14 · Wed Mar 16 2022 00:54:34 GMT+0800 (China Standard Time)

Mine is 15 second (in homebridge-nut), with the actual UPS being polled by upsd/nut every 10s over USB. I've not seen any disconnection since the last update, but the WiFi didn't cut off either, judging from the syslog.

vectronic · Answer 15 · Mon May 23 2022 18:13:22 GMT+0800 (China Standard Time)

I have reduced the logging level to avoid polluting the logs. I will close this issue as I believe the disconnect every minute is now an explained behaviour AND the original issue doesn't seem to be occurring again (at least I haven't heard anything).

I don't really want to modify the plugin to automatically disconnect after polling each time as there is a chance I will introduce new issues.

If the issue persists etc. let me know.