Periodic disconnects - add scheduled reboot?
amil109 opened this issue · comments
Hi CircuitSetup,
I'm really enjoying my Split Single Phase Energy meter, with only one issue - every now and then, approximately every 10 days, it disappears from the network, and requires a restart.
I've had other ESP devices in the past that needed a periodic reboot (once / week, etc) to perform reliably - I could never quite get to the bottom of what the issue was, and didn't have the time to resolve it 'properly' either. After reading a few things online, I thought maybe the meter might be swapping APs or something (multiple APs in the house), and having issues, but I've switched it to an SSID that is only on one AP (with only the ESP on the SSID), so it should not be this.
Please note that I have read "Using web interface and/or leaving it open causes ESP32 memory leak/crash" - I'm running v2.5.5, so should not be this either (although I do always use Chrome).
Right now, I've got the meter connected through a smart outlet, so if it goes down I can disconnect and reconnect it remotely, but I would rather have this automated on the ESP itself.
Can you help with solving this issue? I can certainly send logs (let me know how), or, alternatively, could you add a scheduled reboot to the WebUI? I think this would 'solve' the issue.
Has anyone else reported this?
Kind regards,
Andrew
Hi @amil109 - I'm sorry it's giving you issues! It looks like you have the kit that came programmed. When you updated to v2.5.5, did you compile it, or just load the binary via the web interface? I ask because if you compiled and the libraries weren't up to date, that could still cause memory leak issues.
It seems that some people still do have this issue, but it's usually related to the signal strength, and the ESP32 being able to reconnect to the AP when it looses the connection. After trying to reconnect, it will go into AP mode, which uses a lot of power. It can sometimes cause the brownout detector to trip, and the ESP32 to freeze.
What specific AC transformer are you using? Is it at least 1000mA?
Hi @CircuitSetup ,
Thanks for the quick response!
I used the binary to flash via the web interface.
My AC transformer is 2000mA, it's one for 230V, not listed, so I've got a calibrated value in the WebUI.
The ESP is very close to my router (~1m, they're in the same cupboard) I don't know if this might be an issue?
Kind regards,
Andrew
Your transformer is outputting enough current. What is the voltage output? Also is it one that plugs into the wall? Some people have had some issues with low grade transformers.
The closer to ESP32 is to the AP, the better. That's probably not the issue either.
Hi @CircuitSetup ,
It is a 'wall wart' type transformer, 9V AC 2000mA output - this is the link.
If you can recommend a better / different brand for use in the UK, please let me know - I will happily purchase it!
Kind regards,
Andrew Mills
There isn't too much detail about how that transformer works on that page, but I can see that it's passed UK standards, so it's probably doing what it is supposed to do.
In EmonCMS, you can also take the available memory and make that an input, graph it, and see if it running out of memory. If it is there is an unexpected leak somewhere. If not, then it's likely a connection/power issue.
If it is a power issue the only other suggestion I would have before trying a different ESP32, is to put a 10uf to 50uf through hole style capacitor between the 3v3 and gnd pins on the ESP32. This can either be soldered in or put in the pins that hold the ESP32 on the underside of the adapter board.
Let me know what you find.
One other thing - Are you sending any data via MQTT? I just fixed a bug with that when it got disconnected.
Yes - I am sending MQTT data to my OpenHab instance - should I stop this? TBH I'm not using it right now. I'm also happy to flash a Beta with the fix if you have it?
I fixed an issue in the latest release that would cause MQTT to try to reconnect for 14 min straight if the connection was lost. It was only supposed to do this for 5 seconds, then every 10 seconds. This could cause the ESP32 to lock up, which may be what you are experiencing. Please flash the latest bin to the ESP32 via the web interface, and let me know if this helps.
I've flashed it now - I'll keep you posted!
Prior to this, it worked for ~15 days without issue, so it might be a hard one to detect. If I kill my MQTT server and wait, that should replicate it right? I can certainly try that. It's completely possible that the energy monitor failures were aligned with me messing around with the MQTT server, and I just didn't realise in the past.
Hi @CircuitSetup,
My device has been operating without issue for a week since flashing - I think if it lasts one more week without issue we can consider this topic closed! Thanks very much for your help. I'll post if I have any issues in the next 7 days.
Great! I'm glad to hear it's working better for you now.
Hi @CircuitSetup I got my device and it's great. Works really well, besides the disconnects. I thought it was my wifi signal as my AP was very far.. I have since got another AP in the mix ( Unifi AP Setup, network is solid ) and I get a great -40 dBm connection. Yet at random.. It can happen more than once a day, but it's at minimum once a day I will need to unplug the power supply and plug it back in.. If I am running a consistent ping it comes back really fast, 3-5 seconds. Is that even a full boot? Any idea what could be going on? I am only running Emoncms also .. No MQTT from this device.
Free RAM: 213K
Flash Size: 4096K
Version: v2.5.7
Oh , power supply was included in your provided kit. I am using a full kit from you.
@CircuitSetup John, Could you please advise.. It's really annoying this is not a stable device..
Sorry @McGeaverBeaver, I overlooked your initial message.
Did you update the firmware from the provided bin or did you compile from source?
Also, when it randomly disconnects is there a browser window open anywhere?
It may help to collect data for available ram in emoncms to see if the esp is running out, then freezing. That would indicate a memory leak. I've seen this happen when the software isnt closing connections correctly.
@CircuitSetup No problem.. I have updated the FW from the provided bins.. You have 2 , do I need both uploaded? I have done that src.spiffs & src.ino.esp32.. It's on v 2.5.7 and said successful after both uploads..
No browsers opened when this happens. I only have it logging to Emoncms..
I can also check the lights before I reboot it next time this happens but I think the red light is on and no blue light..
is a reboot normal to only take a few seconds? unplug power supply from wall and plug it back in..
You shouldn't need to update the SPIFFS file, no. The latest just has some text changes in the admin interface.
It sounds like the esp32 is freezing. Are you using the transformer that came with the kit or another one?
If you can, I would monitor the available memory, which you should see as an input in emoncms. This could indicate a memory leak.
Ok I have added RAM to start logging.. I will let you know the results .. Currently sitting at 225464
Oh and using the transformer that came with the kit.. Full kit, I have made no modifications to it.
@CircuitSetup Hey , So it happened again, here is a short clip of showing the ram usage .. doesn't look to be memory related.. https://fastshare.space/33d36b3b6RS.mp4
@McGeaverBeaver I'm sorry it's still giving you issues. If it's not memory, the ESP32 is either having issues with power, or staying connected to your AP. Is your network configured with any repeaters where 2 physical APs have the same name? Or 5ghz and 2.5ghz bands with the same name?
Hey @CircuitSetup
So I don't think it will be my WiFi , I have a Unifi setup with 2 of the latest AP Pros. I also have a " IoT " SSID set for only 2.4ghz.. I have over 30 devices on my WiFi and none have issues.
As for power issues , you mean delivery? I can ensure you there are no power issues in my home when this happens..
Any way to get a daily reboot integrated, It seems that would just be the easy fix here as unplugging the transformer from the wall and plugging it back again right away, not waiting.. and if I have a ping up on another system the network comes back in just a few seconds..
so I ask again, how long does this normally take to reboot from full power loss? I really don't feel like to open my panel up and d/c the CT's so I can test this as there looks to be retained power from the CT's..
photos of the device / connection to power
Thanks for clarifying!
I meant to say, regarding power issues, is that the ESP32, for some reason, requires more power than what is available, or there is just something wrong with it. When this happens, it could just be losing the wifi connection over and over, then freezing when trying to reconnect to the AP.
Normally, it should take around 10 seconds to reboot and re-connect to the programmed AP.
The right way to do it, is to set up some kind of watchdog - if the meter doesn't send data for x amount of time. Power cycle the AC adapter via smart plug. But that's still a bandaid.
Either way, I want to try a new (programmed) ESP32. If you can provide your order number, I'll get it mailed out.
Also, nice homelab!
@CircuitSetup Thanks John, Fair enough it would only be a bandaid and not identify root cause..
My Order 89095 from crowd supply. I didn't think about it until last night my Konnected.IO also runs on an ESP32 and it doesn't have the same disconnect issues so maybe you are right and onto something..
Thanks again John!
Oh okay - you must not be in my order system. Please send me your mailing address via this form: https://circuitsetup.us/index.php/contact/
@CircuitSetup My bad.. I sent it twice , Forgot the address the first message. Thanks Again John!
Great, I'll send it out today!
@CircuitSetup Thanks!! I'll give you an update after I swap it out and wait a couple days!
@CircuitSetup Hey John, So I got the new ESP32 , swapped it out and I've def noticed that it's been more stable.. But I've had it disconnect on me twice without recovery.. Should it not remain to try to connect to AP and go back into AP mode every 5 minutes? I don't even see the ESP broadcasting it's SSID right now with the current failure and lights on it are : On your board - Green Light, Red / Blue Light flashing. On the ESP - solid red light, blue light flashing. not sure if that helps you.
@McGeaverBeaver It should try to reconnect to your AP every 5 min if it fails, yes (assuming the ESP32 isn't frozen), and will be in AP mode otherwise.
The blue LED will indicate the status:
- blinks every 2 seconds - the ESP32 is in AP mode, and it should be broadcasting.
- blinks every 4 seconds - connected to your AP.
- doesn't blink at all, then the ESP32 is likely locked up.
I just updated EmonESP to v2.6, which has a number of bug fixes, adds a watchdog timer, and replaces the MQTT library. See details here: https://github.com/CircuitSetup/Split-Single-Phase-Energy-Meter/releases/tag/v2.6
Edit:
@CircuitSetup So I had it on a relay and was rebooting it if it stopped responding to ping.. Was working well, didn't have any gaps in data for days as the reboots take only a second.
I did as you suggested and updated to - v2.6-src.ino.esp32.bin and I lost Emoncms communcation.
Managed to downgrade back to 2.5.7 and Emoncms is working fine again.. Not sure why it wouldn't work on 2.6.
I'm not sure why you wouldnt be able to access the interface with v2.6. Was the blue light blinking on the esp32? If so, it was either in AP mode or connected to your AP with a new IP. Maybe your AP thought it was a new device?
@CircuitSetup John.. Enough with the help desk troubleshooting.. It seems to me you're not reading also my posts and I'm legit trying to help you fix YOUR product. I have it working in a hacked up way that I'm " Fine " with. You're the one who clearly stated it should not operate this way and you continue to blame my wifi.. I feel you're treating me like a 80 year old grandmother having issues with Facebook Live.. " IS YOUR WIFI ON? "
I appreciate the fact you sent out new hardware and attempted to see if that was the issue.. But it's not. I am running UNIFI AP PRO Access Points, With a Unifi Switch and $600 mini PC running PFSENSE. My network is not the issue of your product. WIFI is not the issue.
Now,. Please read my post again from yesterday and I will re-write it here..
As you suggested - I upgraded to v2.6-src.ino.esp32.bin
the EMONESP Interface is reachable... I never stated it was not..
EMONCMS from your EMONESP interface won't establish a connection with my server. I have tried factory reset, upgrade firmware again, input server/api key from fresh factory reset and connected would remain " NO "
I downgraded back to 2.5.7 and EMONCMS connected fine again.
John.. I ask you to STOP blaming my WIFI or Network, Until you get logs showing me otherwise ..
The disconnecting issue is a result of your software or hardware. Period.
Also, I gotta rant here some more.. How the **** do you think I downgraded to 2.5.7 from 2.6 if I lost access to the device and " my AP thought it was a new device? " lol... C'mon man, What kind of folks you think are buying your devices?
Sorry, I didn't mean to be condescending. Sometimes replying from your phone isn't the best idea. I misread what you were saying, and I forgot that you actually have some good network hardware and know what you're doing.
In my tests I haven't had issues connecting to EmonCMS, so I must be missing something. There were a couple things related to how it's connecting to EmonCMS that were changed in v2.6:
- The lib that http and https uses was changed since the old one was being used incorrectly.
- There's now a timeout if it doesn't connect within 4 seconds to accomodate for the watchdog.
- Some String's were changed to char's to prevent memory issues
Are you using http or https to connect with EmonCMS?
I thought you downgraded by connecting the ESP32 to your computer and uploaded the bin.
Chiming in here, 2.6 has definitely been more stable but I've had a couple more disconnects this week. I'm using a variety of other ESP32 firmware (OpenGarage/WLED/ESPHome) without issues, and my AC transformer is a 1.5A 9V Jameco. A mild annoyance for an otherwise great product!
@genehand I'm sorry its still giving you issues. I believe the cause is a combination of being able to stay connected to an AP, and the error handling around that. I'll continue to improve the software.
By the way, you can load ESPHome on the meter esp32 to see if that improves stability. Details here: https://next.esphome.io/components/sensor/atm90e32.html
I'm having issues still as well. RSSI is -72 typically. I'm on 2.6. I'll go check on it after a few days, and it'll be frozen, so it's missing data. Hard for me to calculate monthly costs when I'm missing multiple days at a time. I've had this issue since purchasing it. I've gotten to the point that I don't even rely on it anymore.
@scfigg yep still an issue.. He mentioned my WiFi also.. I went out and purchased another AP ( A Unifi AP ) - It's now VERY close to the device, on it's own SSID 2.4 broadcasting only from that AP with all the options toggled around to see if that would change stability.. I also have a Konnected Alarm Panel and it uses the same ESP wifi chip and I have NO issues with this device.
It's software issue here.
My fix, I put a script to ping it on HA and if it can't reach it , it cycles a smart switch I have the transformer of the energy meter plugged into.. The smart switch does not change readings, and the reboot is so fast that the data loss for that few seconds is not really a big deal..
Thanks. I was actually just contemplating both of those things - whether or not to put an AP right next to it (I have some nice Ubiquiti network setup in my house, so signal is generally not an issue). I was also thinking of plugging a smart switch into it and forcing reboots.
Care to share your HA scripts for ping/reboot when you get a chance? I'm sure I could figure it out, but if you have something already setup, that would be much easier.
Yea man, WiFi not the problem here..
Let me get what I'm doing for the reboots.. I don't recall from the top of my head also what I am doing lol
@scfigg lol it works but man I was lazy! I just have a ping sensor setup in HA , and then an automation that checks every 5 minutes to see if the ping is still alive, if it's not.. toggle power.. I had it on an actual TOGGLE before but I find that for some reason things would be on or off when I didn't want them so I just have it toggle but with an ON/OFF
configuration.yaml
- platform: ping
name: emonesp_wifi
host: 192.168.99.248
scan_interval: 60
automations.yaml
- id: '123456'
alias: Toggle emon ESP
description: ''
trigger:- minutes: /5
platform: time_pattern
condition: - condition: state
entity_id: binary_sensor.emonesp_wifi
state: 'off'
action: - data: {}
entity_id: switch.emondumbpower
service: switch.turn_off - delay: 00:00:02
- data: {}
entity_id: switch.emondumbpower
service: switch.turn_on
- minutes: /5
Thank you! That saved me a bunch of time :) I'll give this a run and see how it holds up. Thank you! Now just to figure out how to estimate monthly energy bill cost in HA!
I'm sorry you guys are still having issues with connectivity. I am working on a new version of EmonESP, that hopefully will remedy these issues. There's a lot of things that have been re-worked, and there's a lot of new features too.
In the meantime, since I see Home Assistant is being used, have you considered trying ESPHome?
No worries John, The workaround is easy enough and reporting has been great since.
Yes I have briefly looked at ESPHome and I understand with this we can use to identify single devices out being on ESPHome, but I honestly don't know where to start and haven't looked very hard.. I did enjoy the ease of plug and play with your device.
Looking forward to the update! thanks again John.
Not sure if this helps, but I built a very slimmed down version of the code straight on an ESP32 with almost the only external library being wifimanager. I used it in such a way that it would put the device in AP mode if it ever lost connection.
From what I could see, every few days the device would stop broadcasting data and go into AP mode, so maybe there is an issue in the Wifi library, just an idea, maybe it helps, maybe not.
Any updates yet above 2.6? Still having random lock up's and freezes on the Solar Setup? Are you still supporting this?