MQTT disconnecting while network seems to be OK
Rainerlan opened this issue · comments
I have the following strange behavior and wonder what the problem might be - maybe a bug - maybe a network issue of some kind?
At 1st I'd suspected my network - but really I don't know - Maybe anyone has seen similar problems and found a solution to it?
Environment
I am using Homeassistant and the mosquitto addon on a raspi 4, 64bit bullseye
The problem is very evident on one thermostat only (1 out of 8) - while I see similar behavior on others - but only occasionally.
So this is why I thought that might be a network or HA performance issue - but when investigating in the problem, I find a last will message - For no obvious reason (network problem).
Investigation
I am using the following command on a 2nd machine to listen to all published messages on thermostat_5693070 - The one showing the problem very often:
mosquitto_sub -h 192.168.178.111 -u <username> -P <passwd> -p 1883 -t thermostat_5693070/# | xargs -d$'\n' -L1 bash -c 'date "+%Y-%m-%d %T.%3N $0"'
Result
As one can see on the console output, for a while the thermostat replies on the (through HA) requested changes (schedulesMode & DeviceOn) right away - until a last will message (at 2022-12-26 21:19:21.209) can be found (send by the broker as I understand now).
2022-12-26 21:19:21.209 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":false}
After the thermostat reconnects to the MQTT broker, the requested changes are being answered (seems like the commands had been queued).
I.e. the the problem is, the connection between thermostat and broker seems to be broken.
But I cannot see any network issues or disconnect-messages in the mqtt brokers logs.
Note: Through http, the thermostat is always reachable - and a ping being run in parallel shows perfect response-times...Even when the thermostat finds itself diconnected:
MQTT-log, listening to the thermostat
2022-12-26 21:13:44.425 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:13:44.441 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:13:44.491 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:13:45.241 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:13:45.257 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:13:45.517 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:13:46.225 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:13:46.240 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:13:46.365 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:13:47.530 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:13:47.540 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:13:47.802 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:13:48.639 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:13:48.645 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:13:48.822 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:19:21.209 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":false}
2022-12-26 21:20:39.535 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true}
2022-12-26 21:23:29.648 thermostat_5693070/clock/properties {"ntpServer":"pool.ntp.org","timeZoneServer":"http://worldtimeapi.org/api/ip","epochTime":1672089809,"epochTimeFormatted":"2022-12-26 21:23:29","validTime":true,"timezone":"Europe/Berlin"}
2022-12-26 21:23:48.867 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:24:21.198 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:21.242 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:24:26.488 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:26.504 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:24:27.785 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:27.795 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:24:28.423 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:28.432 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:24:28.865 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:28.875 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:24:31.137 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:31.151 thermostat_5693070/thermostat/set/schedulesMode off
2022-12-26 21:24:31.634 thermostat_5693070/thermostat/set/deviceOn True
2022-12-26 21:24:31.647 thermostat_5693070/thermostat/set/schedulesMode auto
2022-12-26 21:25:08.178 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:25:08.365 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:25:08.474 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:25:08.577 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:25:08.683 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:25:08.789 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"auto","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
2022-12-26 21:26:01.077 thermostat_5693070/thermostat/properties {"idx":"therm5693070","ip":"192.168.178.135","alive":true,"firmware":"1.1.1 Beta","Temperature":22.00,"TargetTemperature":22.00,"FloorTemperature":0.00,"deviceOn":true,"schedulesMode":"off","freezeMode":"off","locked":false,"Heater":false,"MinHeaterTemperature":5.00,"MaxHeaterTemperature":35.00,"TemperatureCorrection":0.00,"SwitchDifference":1.00,"sensorSelection":"internal"}
ping times
Those are the ping times when the thermostat finds itself disconnected from MQTT broker - Imho nice ping times...
64 bytes from 192.168.178.135: icmp_seq=21 ttl=255 time=10.5 ms
64 bytes from 192.168.178.135: icmp_seq=22 ttl=255 time=4.36 ms
64 bytes from 192.168.178.135: icmp_seq=23 ttl=255 time=6.65 ms
64 bytes from 192.168.178.135: icmp_seq=24 ttl=255 time=7.12 ms
64 bytes from 192.168.178.135: icmp_seq=25 ttl=255 time=7.70 ms
64 bytes from 192.168.178.135: icmp_seq=26 ttl=255 time=6.32 ms
64 bytes from 192.168.178.135: icmp_seq=27 ttl=255 time=5.60 ms
64 bytes from 192.168.178.135: icmp_seq=28 ttl=255 time=4.95 ms
64 bytes from 192.168.178.135: icmp_seq=29 ttl=255 time=4.34 ms
Anyone having an idea how to investigate the problem further?
Possible workaround
What I found when looking at the code is, that the reconnect is "only" tried every 5min (300000ms).
So to improve the situation, I could just lower the re-connection time to eg. every 10s (10000ms).
That of course won't fix the connection losses - but will conceal the problem.
Some additional insights...
I turned logging on in mosquitto and found at least one strange log message for the thermostat in question:
2022-12-26 23:32:01: Client thermostat_5693070 disconnected due to protocol error.
Yet I am not sure what could be the reason for this...(If this is a problem with mosquitto or the thermostat)
I've had funny symthons when using the same mqtt client id in multiple scripts.
But I guess "thermostat_5693070" is yours and it looks pretty device specific?
Hi, the actual reason was, that Wifi was to busy to transmit the MQTT ping requests/responses in time - So the broker timed out the connection.
After having moved all broker-side network traffic to Ethernet (from Wifi - by chnaging the IP in the MQTT settings of the thermostats), the responses (transmit delay) improved and the disconnects got much better! Now they are very sporadic - sometimes once a day.
Also I increased MQTT_KEEPALIVE to 30 - I think a 15s ping req/resp is not needed at all. Even 60s would be sufficient.