tuyaface spins out of control
TradeFace opened this issue · comments
As discussed here TradeFace/tuyagateway#14
tuyamqtt | 2020-05-27 19:29:46,420 WARNING (Thread-66) [tuyaface.tuyaclient] (192.168.1.21) reconnecting
tuyamqtt | 2020-05-27 19:29:46,420 INFO (Thread-66) [tuyaface] (192.168.1.21) Connecting to 192.168.1.21
tuyamqtt | 2020-05-27 19:29:46,422 INFO (Thread-66) [tuyaface.tuyaclient] (192.168.1.21) connected
tuyamqtt | 2020-05-27 19:29:46,431 WARNING (Thread-66) [tuyaface.tuyaclient] (192.168.1.21) reconnecting
tuyamqtt | 2020-05-27 19:29:46,431 INFO (Thread-66) [tuyaface] (192.168.1.21) Connecting to 192.168.1.21
tuyamqtt | 2020-05-27 19:29:46,433 INFO (Thread-66) [tuyaface.tuyaclient] (192.168.1.21) connected
tuyamqtt | 2020-05-27 19:29:46,442 WARNING (Thread-66) [tuyaface.tuyaclient] (192.168.1.21) reconnecting
tuyamqtt | 2020-05-27 19:29:46,442 INFO (Thread-66) [tuyaface] (192.168.1.21) Connecting to 192.168.1.21
going wild! Unclear what the cause is. But it seems to overflow the device in HA. (available vs unavailable) The graph for the device can't be loaded after that.
It seems to go wild after No reply to status
2020-05-30 23:10:35,875 WARNING (bfab167947172eb193wwc7) [tuyaface.tuyaclient] (192.168.1.20) No reply to status
2020-05-30 23:10:36,333 WARNING (Thread-58) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
2020-05-30 23:10:36,333 INFO (Thread-58) [tuyaface] (192.168.1.20) Connecting to 192.168.1.20
2020-05-30 23:10:36,382 INFO (Thread-58) [tuyaface.tuyaclient] (192.168.1.20) connected
2020-05-30 23:10:36,395 WARNING (Thread-58) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
2020-05-30 23:10:36,396 INFO (Thread-58) [tuyaface] (192.168.1.20) Connecting to 192.168.1.20
2020-05-30 23:10:36,397 INFO (Thread-58) [tuyaface.tuyaclient] (192.168.1.20) connected
2020-05-30 23:10:36,406 WARNING (Thread-58) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
OK, which commit do you get this on?
this is on
tuyamqtt$ git status
On branch development
Your branch is up to date with 'origin/development'.
tuyamqtt/tuya$ git status
On branch master
Your branch is ahead of 'origin/master' by 31 commits.
Oops... forgot to checkout. I'll monitor it some more on dev. See if it occurs again.
you can do git log -n 1 to show the exact commit you're on.
tuyamqtt/tuya$ git log -n 1
commit e457562bdb5fef9e64e122677973e1f53cfe46c8 (HEAD -> development, origin/development, master
tuyamqtt$ git log -n 1
commit 2f83b230a103611684590cb308c3ac15d299cea5 (HEAD -> development, origin/development)
Did you get the error on that commit, or are you on that commit after updating the branch though?
this was after I did the checkout on tuyaface. Vague answer... So let's try that again. When I reported earlier today, I was on master. Because I forgot to do a checkout to development. After your comment to check the commit I posted the stuff then changed to the development branch.
In short I got the error on the commit below
tradeface@media:/var/docker/homeassistant/tuyamqtt/tuya$ git checkout master
Switched to branch 'master'
Your branch is ahead of 'origin/master' by 31 commits.
(use "git push" to publish your local commits)
tradeface@media:/var/docker/homeassistant/tuyamqtt/tuya$ git log -n 1
commit e457562bdb5fef9e64e122677973e1f53cfe46c8 (HEAD -> master, origin/development, development)
This below was triggered by saving a device in GismoCaster. This probably not a tuyaface problem, a tuyamqtt one. In the log below there are multiple threads for the same ip, instead of the expected two. i.e. the filter isn't working.
2020-05-31 00:42:46,535 INFO (Thread-109) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
b'{"name": "switch hallway 3 gang", "deviceid": "bfe32bb5914f3d7ff6xzts", "localkey": "bc560885ae67800b", "ip": "192.168.1.31", "protocol": "3.3", "pref_status_cmd": 13, "ha_discovery": true, "tuya_discovery": true, "dps": [{"key": 1, "name": "Landing Lighting Hotel"}, {"key": 2, "name": "Hallway Lighting"}, {"key": 3, "name": "Frontdoor Lighting"}]}'
tuya/discovery/bfe32bb5914f3d7ff6xzts {'name': 'switch hallway 3 gang', 'deviceid': 'bfe32bb5914f3d7ff6xzts', 'localkey': 'bc560885ae67800b', 'ip': '192.168.1.31', 'protocol': '3.3', 'pref_status_cmd': 13, 'ha_discovery': True, 'tuya_discovery': True, 'dps': [{'key': 1, 'name': 'Landing Lighting Hotel'}, {'key': 2, 'name': 'Hallway Lighting'}, {'key': 3, 'name': 'Frontdoor Lighting'}]}
2020-05-31 00:42:46,535 INFO (Thread-1) [tuyamqtt] discovery message received {"name": "switch hallway 3 gang", "deviceid": "bfe32bb5914f3d7ff6xzts", "localkey": "bc560885ae67800b", "ip": "192.168.1.31", "protocol": "3.3", "pref_status_cmd": 13, "ha_discovery": true, "tuya_discovery": true, "dps": [{"key": 1, "name": "Landing Lighting Hotel"}, {"key": 2, "name": "Hallway Lighting"}, {"key": 3, "name": "Frontdoor Lighting"}]} topic tuya/discovery/bfe32bb5914f3d7ff6xzts retained 0
2020-05-31 00:42:46,536 INFO (Thread-108) [tuyamqtt] MQTT Connection state: Connection successful for tuya/bf64542d9b17f9abedirsa
2020-05-31 00:42:46,537 INFO (Thread-112) [tuyaface] (192.168.1.31) Connecting to 192.168.1.31
2020-05-31 00:42:46,537 INFO (Thread-111) [tuyamqtt] MQTT Connection state: Connection successful for tuya/bfe32bb5914f3d7ff6xzts
2020-05-31 00:42:46,538 INFO (Thread-109) [tuyaface.tuyaclient] (192.168.1.25) connected
2020-05-31 00:42:46,540 ERROR (Thread-109) [tuyaface.tuyaclient] (192.168.1.25) exception when reading from socket
2020-05-31 00:42:46,541 ERROR (Thread-86) [tuyaface.tuyaclient] (192.168.1.25) exception when reading from socket
2020-05-31 00:42:46,549 ERROR (Thread-97) [tuyaface.tuyaclient] (192.168.1.25) exception when reading from socket
2020-05-31 00:42:46,574 INFO (Thread-79) [tuyaface.tuyaclient] (192.168.1.20) connected
2020-05-31 00:42:46,576 INFO (Thread-91) [tuyaface.tuyaclient] (192.168.1.20) connected
2020-05-31 00:42:46,577 INFO (Thread-103) [tuyaface.tuyaclient] (192.168.1.20) connected
2020-05-31 00:42:46,578 ERROR (Thread-91) [tuyaface.tuyaclient] (192.168.1.20) exception when reading from socket
2020-05-31 00:42:46,578 ERROR (Thread-103) [tuyaface.tuyaclient] (192.168.1.20) exception when reading from socket
2020-05-31 00:42:46,595 INFO (Thread-77) [tuyaface.tuyaclient] (192.168.1.31) connected
2020-05-31 00:42:46,596 INFO (Thread-88) [tuyaface.tuyaclient] (192.168.1.31) connected
2020-05-31 00:42:46,596 ERROR (Thread-79) [tuyaface.tuyaclient] (192.168.1.20) exception when reading from socket
2020-05-31 00:42:46,596 INFO (Thread-100) [tuyaface.tuyaclient] (192.168.1.31) connected
2020-05-31 00:42:46,597 INFO (Thread-112) [tuyaface.tuyaclient] (192.168.1.31) connected
2020-05-31 00:42:46,597 INFO (Thread-82) [tuyaface.tuyaclient] (192.168.1.21) connected
2020-05-31 00:42:46,598 INFO (Thread-94) [tuyaface.tuyaclient] (192.168.1.21) connected
2020-05-31 00:42:46,599 INFO (Thread-106) [tuyaface.tuyaclient] (192.168.1.21) connected
2020-05-31 00:42:46,603 ERROR (Thread-106) [tuyaface.tuyaclient] (192.168.1.21) exception when reading from socket
2020-05-31 00:42:46,608 ERROR (Thread-100) [tuyaface.tuyaclient] (192.168.1.31) exception when reading from socket
2020-05-31 00:42:46,610 ERROR (Thread-82) [tuyaface.tuyaclient] (192.168.1.21) exception when reading from socket
2020-05-31 00:42:46,615 ERROR (Thread-112) [tuyaface.tuyaclient] (192.168.1.31) exception when reading from socket
2020-05-31 00:42:46,617 ERROR (Thread-94) [tuyaface.tuyaclient] (192.168.1.21) exception when reading from socket
2020-05-31 00:42:46,749 WARNING (Thread-29) [tuyaface] (192.168.1.27) Failed to connect to 192.168.1.27. Retry in 1 seconds
2020-05-31 00:42:46,749 ERROR (Thread-29) [tuyaface.tuyaclient] (192.168.1.27) exception when opening socket
2020-05-31 00:42:46,784 WARNING (Thread-49) [tuyaface] (192.168.1.201) Failed to connect to 192.168.1.201. Retry in 1 seconds
2020-05-31 00:42:46,784 ERROR (Thread-49) [tuyaface.tuyaclient] (192.168.1.201) exception when opening socket
2020-05-31 00:42:48,624 WARNING (bfab167947172eb193wwc7) [tuyaface.tuyaclient] (192.168.1.20) No reply to status
2020-05-31 00:42:48,625 WARNING (bfef014d3f4c40e9b0ogku) [tuyaface.tuyaclient] (192.168.1.21) No reply to status
2020-05-31 00:42:48,626 WARNING (bf64542d9b17f9abedirsa) [tuyaface.tuyaclient] (192.168.1.25) No reply to status
2020-05-31 00:42:48,628 WARNING (bfab167947172eb193wwc7) [tuyaface.tuyaclient] (192.168.1.20) No reply to status
2020-05-31 00:42:48,630 WARNING (bfef014d3f4c40e9b0ogku) [tuyaface.tuyaclient] (192.168.1.21) No reply to status
2020-05-31 00:42:48,631 WARNING (bf64542d9b17f9abedirsa) [tuyaface.tuyaclient] (192.168.1.25) No reply to status
2020-05-31 00:42:48,631 WARNING (bfe32bb5914f3d7ff6xzts) [tuyaface.tuyaclient] (192.168.1.31) No reply to status
2020-05-31 00:42:48,633 WARNING (bfab167947172eb193wwc7) [tuyaface.tuyaclient] (192.168.1.20) No reply to status
2020-05-31 00:42:48,633 WARNING (bfef014d3f4c40e9b0ogku) [tuyaface.tuyaclient] (192.168.1.21) No reply to status
2020-05-31 00:42:48,635 WARNING (bf64542d9b17f9abedirsa) [tuyaface.tuyaclient] (192.168.1.25) No reply to status
2020-05-31 00:42:48,637 WARNING (bfe32bb5914f3d7ff6xzts) [tuyaface.tuyaclient] (192.168.1.31) No reply to status
2020-05-31 00:42:49,042 WARNING (Thread-109) [tuyaface.tuyaclient] (192.168.1.25) reconnecting
2020-05-31 00:42:49,042 WARNING (Thread-86) [tuyaface.tuyaclient] (192.168.1.25) reconnecting
2020-05-31 00:42:49,043 INFO (Thread-109) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
2020-05-31 00:42:49,043 INFO (Thread-86) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
2020-05-31 00:42:49,051 WARNING (Thread-97) [tuyaface.tuyaclient] (192.168.1.25) reconnecting
2020-05-31 00:42:49,051 INFO (Thread-97) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
2020-05-31 00:42:49,071 INFO (Thread-86) [tuyaface.tuyaclient] (192.168.1.25) connected
2020-05-31 00:42:49,072 INFO (Thread-109) [tuyaface.tuyaclient] (192.168.1.25) connected
2020-05-31 00:42:49,072 INFO (Thread-97) [tuyaface.tuyaclient] (192.168.1.25) connected
2020-05-31 00:42:49,081 WARNING (Thread-91) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
2020-05-31 00:42:49,081 INFO (Thread-91) [tuyaface] (192.168.1.20) Connecting to 192.168.1.20
2020-05-31 00:42:49,083 WARNING (Thread-103) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
2020-05-31 00:42:49,083 INFO (Thread-103) [tuyaface] (192.168.1.20) Connecting to 192.168.1.20
2020-05-31 00:42:49,085 WARNING (Thread-86) [tuyaface.tuyaclient] (192.168.1.25) reconnecting
2020-05-31 00:42:49,085 INFO (Thread-86) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
2020-05-31 00:42:49,096 WARNING (Thread-109) [tuyaface.tuyaclient] (192.168.1.25) reconnecting
2020-05-31 00:42:49,096 INFO (Thread-109) [tuyaface] (192.168.1.25) Connecting to 192.168.1.25
2020-05-31 00:42:49,099 WARNING (Thread-79) [tuyaface.tuyaclient] (192.168.1.20) reconnecting
2020-05-31 00:42:49,099 INFO (Thread-79) [tuyaface] (192.168.1.20) Connecting to 192.168.1.20
I did some extra checks in tuyamqtt. That will prevent most cases, hopefully. However we can't be sure that no other program will try to access the tuya devices. i.e. a split brain without a single source of truth. The logs above don't tell us which path is taken to come to a reconnect. In case _is_connection_stale the force_sleep isn't set, which might explain why it is spinning connections like crazy.
PR #70 adds some extra prints to help make it clear why it's reconnecting so quickly.
Can you reproduce the bug with those prints added?
If I run 2 instances of tuyamqtt and send in a command I think I should be able to reproduce. Will try in a bit.
Well 2 instances was all it took.
https://gist.github.com/TradeFace/349caa37103407a2f6d58f7395f78873
tuyamqtt$ git log -n 1
commit a8386248cfa18cf60461b98047947f3dbbb9627a (HEAD -> development, origin/development)
Merge: 2f83b23 d22f15e
Date: Sun May 31 10:09:37 2020 +0200
Merge pull request #59 from TradeFace/fix/devicefilter
Fix/devicefilter #57
tuyamqtt/tuya$ git log -n 1
commit 9d7916e79e7dddb6cfd9a798382bc433e6f0442e (HEAD -> development, origin/development)
Date: Sun May 31 10:31:51 2020 +0200
default preferred status command
Thanks for the log! The reason is clear, I'll push a fix tomorrow.