Socket Timeouts
bdunlay opened this issue · comments
I get a socket timeout about once a minute when accessing the Lovelace UI through HAProxy. This does not repro when accessing the local server directly. It can be mitigated by increasing the http tunnel timeout in HAproxy to 60 seconds. However, this is likely to affect other configurations too, such as Nginx.
I wonder if this could be mitigated natively by sending a periodic ping to keep the connection active (so it doesn't stagnate), perhaps every 5-15 seconds.
It's a problem for users because it causes the whole page to refresh once the socket connection times out. If you're working on the UI or an Automation, the form is lost in the refresh while a connection is re-established.
E.g. trivial example here https://www.jstips.co/en/javascript/working-with-websocket-timeout/
We already set a heartbeat on the server side at https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/websocket_api/http.py#L111
And browsers should automatically send ping/pong frames, but I guess they might not?
Home Assistant already supports a ping message.
We should initialize and clear the old one here: https://github.com/home-assistant/home-assistant-js-websocket/blob/master/lib/connection.ts#L97
We should reset time till sending a ping message here:
https://github.com/home-assistant/home-assistant-js-websocket/blob/master/lib/connection.ts#L242-L247
55 is not based on anything. Just didn't think that we should cause more traffic than necessary. Do you know what HAProxy and NGINX base their defaults on ?
I did some research, and I think I have a new conclusion about what to do here...
NGINX defaults to 60s, while HAProxy doesn't have a static tunnel timeout default. If tunnel timeout is not defined, it uses either the client or server timeout depending on where they are defined in the configuration file.
From HAProxy source code, in file src/mux_h2.c, on version 2.0.12:
...
547 if (conn_is_back(conn)) {
548 h2c->flags = H2_CF_IS_BACK;
549 h2c->shut_timeout = h2c->timeout = prx->timeout.server; // <---
550 if (tick_isset(prx->timeout.serverfin))
551 h2c->shut_timeout = prx->timeout.serverfin;
552 } else {
553 h2c->flags = H2_CF_NONE;
554 h2c->shut_timeout = h2c->timeout = prx->timeout.client; // <---
555 if (tick_isset(prx->timeout.clientfin))
556 h2c->shut_timeout = prx->timeout.clientfin;
557 }
...
My socket connection appeared to be reconnecting every 50 seconds. Well, looking at my configuration, duh... in fact, the example HAProxy configuration in HomeAssistant docs has timeouts defined in the default section at 50 seconds. That's where I got my configuration from.
...
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 5s #protection from Slowloris attacks
...
So perhaps the real solution here is to just increase the timeouts in the documentation's example HAProxy configuration to 60 seconds to match NGINX, and add a note about why this is important for WebSockets.
Ha, you went deep! Yes, let's just update the docs instead.
Submitted a PR to update the docs. Thanks for getting back to me on this issue!