Huzzah32 V2 + I2C sensor data no longer getting recorded
mswelker opened this issue · comments
Describe the bug
After being initially setup with WhipperSnapper, then configured and recording data on Adafruit IO for some period of time, data stops being recorded on Adafruit IO even though devices are recognized as connected and i2C scans still see the components as connected
Arduino board
Adafruit ESP32 Feather V2 - 8MB Flash + 2 MB PSRAM - STEMMA QT
To Reproduce
- Setup a Feather ESP32 V2 Device on Adafruit IO
- Set wifi values for secrets file and installed Whippersnapper via web serial ( devices report using v1.0.0-beta.63)
- Once device was setup I added the PSMA003l and SHT3C components. Each value from the component was set to 30sec updates, only value not set up for reporting is the Degrees Celcius value from the SHT3C components
- Let device run
Expected behavior
It was expected for data to record every 30 seconds, and for a no-code solution it was. However after seemingly random periods of time, the data completely stopped being recorded (nothing logged, not even NaN values) despite the devices being recognized as online and any i2c scans run still had the PSMA003l and SHT3C components connected (or not when I disconnected the sensors for troubleshooting)
Which components are connected to your device
PMSA003l (PM10, PM25, PM100 Values)
SHT3C (Temp in °F and Humidity Values)
Desktop (please complete the following information):
- Computer: MacBook Pro 2009 i7
- OS: macOS 13.4 (22F66)
- Browser Google Chrome
- Version 114.0.5735.133
Additional context
2000 mAh Lithium Ion Battery connected directly to ESP32 board as a power backup when nut connected to a USB C power supply, but believe devices were plugged in when data stopped being logged
I set up multiple devices with the same configuration to monitor 3 separate areas. The first device I setup recorded data from June 2nd to June 16th, a second and third device recorded data from June 15th to June 20th before data wasn't being recorded. A 4th ESP32 V2 Device was also setup to verify sensors were working using a different wifi connection but it too stopped recording data to Adafruit IO after just a couple of hours.
I have tried to power cycling the boards and while they would reconnect would not record data. Later I tried to erase and do factory resets of the ESP 32 V2 boards, verifying that the default software was working before going back over the online install of WhipperSnapper. After the board is programmed and reset, the device is recognized by Adafruit IO but like with the power resets no new data is recognized or recorded.
I also setup a ESP32 S2 Qt Py to run one set of the PSMA003l and SHT3C components on June 21st and I have not had any issues with it so far, about 50 hours later, but I am keeping an eye on it to see if its data stops getting recorded as well .
@mswelker Seems like the devices aren't dropping offline after a number (2-6) of days but are not sending data to Adafruit IO.
I have tried to power cycling the boards and while they would reconnect would not record data.
Are you able to connect these boards to a serial monitor (or https://www.serialterminal.com/) to get the debug output from them? If you're able to - please post it as a reply to this issue.
When a device with components reconnects to Adafruit IO, IO sends a sensor configuration packet (we call this a sync packet). I think what's happening in these cases is the sync packet(s?) is getting dropped for whatever reason.
Here's the screenshots of serial monitor outputs from three of the Huzzah ESP32 V2 boards I'm having the issue with. I still haven't seen this problem with the ESP32 S2 Qt Py I swapped in to get some data, but have seen the same problem with an ESP32-S2 Feather with BME280 Sensor (screenshot included). I also updated the firmware on one of the V2 boards to v1.0.0-beta.64 and the problem persists.
Unit 1
Unit 7
Unit 2
ESP32-S2 Feather with BME280
Seems like it does receive the I2C message which is 74 bytes but the device is unable to decode it.
Could you paste a screenshot of the device page (on adafruit.io) below for any of those units? I'm going to attempt to replicate this setup.
Unit 7
Unit 7 Device Page
Nice enclosure! I'm going to look at this in-depth after the upcoming USA Holiday (July 4th).
Confirming I am able to replicate this with the same setup
11:49:48.426 -> 2 bytes.
11:49:48.426 -> decodeSignalMsg
11:49:48.426 -> cbSignalMsg
11:49:48.426 -> Sub-messages found: 1
11:49:48.426 -> Signal Msg Tag: Pin Configuration
11:49:48.426 -> Initial Pin Configuration Complete!
11:49:48.426 -> Publishing to pin config complete...
11:49:48.464 -> * NEW MESSAGE [Topic: Signal-I2C]:
11:49:48.464 -> 73 bytes.
11:49:48.464 -> ERROR: Unable to decode I2C message
11:49:48.966 -> Dropped a packet
11:49:49.468 -> Hardware configured successfully!
11:49:51.041 -> Registration and configuration complete!
11:49:51.041 -> Running application...
11:49:51.041 -> PING!
11:49:55.575 -> PING!
11:50:00.126 -> PING!
11:50:04.690 -> PING!
@mswelker I made a small tweak to the MQTT library we use and everything looks OK to me now
Could you please try re-installing WipperSnapper Beta 64? Please let me know if the updated firmware is still having issues for your configuration.
Log:
12:36:37.337 -> * NEW MESSAGE [Topic: Signal-I2C]:
12:36:37.337 -> 77 bytes.
12:36:37.374 -> cbDecodeSignalRequestI2C
12:36:37.374 -> I2C Device LIST Init Request Found!
12:36:37.374 -> EXEC: cbDecodeI2CDeviceInitRequestList
12:36:37.374 -> EXEC: New I2C Port
12:36:37.374 -> Port #: 0
12:36:37.374 -> SDA Pin: 22
12:36:37.374 -> SCL Pin: 20
12:36:37.374 -> Frequency (Hz): 100000
12:36:37.515 -> Attempting to initialize I2C device: pmsa003i
12:36:38.511 -> PM2.5 AQI Sensor Initialized Successfully!
12:36:38.511 -> Publishing Message: I2CResponse...Published!
12:36:38.545 -> EXEC: cbDecodeI2CDeviceInitRequestList
12:36:38.545 -> Attempting to initialize I2C device: shtc3
12:36:38.545 -> SHTC3 Initialized Successfully!
12:36:38.545 -> Publishing Message: I2CResponse...Published!
12:36:38.583 -> Hardware configured successfully!
12:36:40.119 -> Registration and configuration complete!
12:36:40.119 -> Running application...
12:36:40.119 -> PING!
12:36:44.671 -> PING!
@brentru I was able to reinstall WhipperSnapper Beta 64 on one of the Huzzah ESP32 V2 boards and the ESP32-S2 with BME280 board I had problems with. Both of the boards I updated are able to collect and report data to Adafruit IO again and no problems have reappeared in the hour or so I was able to leave them running this afternoon.
I won't be able to set my units back up to let them run continuously again until later this week, but it looks promising so far, thank!
- NEW MESSAGE [Topic: Signal-I2C]:
77 bytes.
cbDecodeSignalRequestI2C
I2C Device LIST Init Request Found!
EXEC: cbDecodeI2CDeviceInitRequestList
EXEC: New I2C Port
Port #: 0
SDA Pin: 22
SCL Pin: 20
Frequency (Hz): 100000
Attempting to initialize I2C device: pmsa003i
PM2.5 AQI Sensor Initialized Successfully!
Publishing Message: I2CResponse...Published!
EXEC: cbDecodeI2CDeviceInitRequestList
Attempting to initialize I2C device: shtc3
SHTC3 Initialized Successfully!
Publishing Message: I2CResponse...Published!
Hardware configured successfully!
Registration and configuration complete!
Running application...
PING!
PING!
PING!
PING!
Sensor 0x12
PM1.0: 16.00 ppm
Sensor 0x12
PM2.5: 25.00 ppm
Sensor 0x12
PM10.0: 25.00 ppm
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
Sensor 0x70
Ambient Temp.: 79.62°F
Sensor 0x70
Humidity: 62.29%RH
PUBLISHING -> I2C Device Sensor Event Message...PUBLISHED!
@tyeth and @brentru I recently had to update my units because of the SSL cert issues last week. A couple of my units seem to be doing fine with WhipperSnapper Beta 66, however it looks like a couple of the units that now have WhipperSnapper Beta 67 are showing the dropped packet problem again. I looked at Issue #452 and tried removing each i2c component, as well as both components and seems like the error persists.
Here's the serial output log from Unit 1 with the SHTC3 and PSMA003I both connected to the Feather Huzzah32 V2 board:
14:38:04.158 -> PING!
14:38:08.290 -> ..
14:38:08.290 -> MQTT topics...
14:38:08.290 -> Running Network FSM...
14:38:08.290 -> Connecting to WiFi.PI_FAST_FLASH_BOOT)
14:38:08.290 -> configsip: 271414342, SPIWP:0xee
14:38:08.290 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
14:38:08.290 -> mode:DIO, clock div:1
14:38:08.290 -> load:0x3fff0030,len:1184
14:38:08.290 -> load:0x40078000,len:13260
14:38:08.290 -> load:0x40080400,len:3028
14:38:08.290 -> entry 0x400805e4
14:38:08.290 -> Adafruit..
14:38:12.440 -> Attempting to connect to WiFi...
14:38:17.952 -> Connected to WiFi!
14:38:17.952 -> Attempting to connect to IO...
14:38:19.799 -> Registering hardware with WipperSnapper...
14:38:19.799 -> Registering hardware with IO...
14:38:19.799 -> Encoding registration request...Encoding registration msg...Published!
14:38:19.799 -> Polling for registration message response...2
14:38:20.386 -> GOT Registration Response Message:
14:38:20.386 -> Hardware Response Msg:
14:38:20.386 -> GPIO Pins: 12
14:38:20.386 -> Analog Pins: 7
14:38:20.386 -> Reference voltage: 3.30v
14:38:20.419 -> Completed registration process, configuration next!
14:38:20.419 -> Polling for message containing hardware configuration...
14:38:20.453 -> Polling for message containing hardware configuration...
14:38:20.488 -> Polling for message containing hardware configuration...
14:38:20.488 -> cbSignalTopic: New Msg on Signal Topic
14:38:20.488 -> 2 bytes.
14:38:20.488 -> decodeSignalMsg
14:38:20.488 -> cbSignalMsg
14:38:20.488 -> Sub-messages found: 1
14:38:20.488 -> Signal Msg Tag: Pin Configuration
14:38:20.488 -> Initial Pin Configuration Complete!
14:38:20.488 -> Publishing to pin config complete...
14:38:20.488 -> * NEW MESSAGE [Topic: Pixels]:
14:38:20.488 -> 16 bytes.
14:38:20.520 -> [Message Type]: wippersnapper_signal_v1_PixelsRequest_req_pixels_create_tag
14:38:20.520 -> Created NeoPixel strand of length 1 on GPIO #D12
14:38:20.520 -> -> wippersnapper_signal_v1_PixelsResponse...* NEW MESSAGE [Topic: Pixels]:
14:38:20.520 -> 13 bytes.
14:38:20.520 -> [Message Type]: wippersnapper_signal_v1_PixelsRequest_req_pixels_write_tag
14:38:20.520 -> Filling color: 312320
14:38:20.558 -> * NEW MESSAGE [Topic: Signal-I2C]:
14:38:20.558 -> 76 bytes.
14:38:20.558 -> ERROR: Unable to decode I2C message
14:38:21.897 -> Dropped a packet
14:38:21.897 -> Published!
14:38:22.070 -> Hardware configured successfully!
14:38:23.965 -> Registration and configuration complete!
14:38:23.965 -> Running application...
14:38:23.965 -> PING!
14:38:28.183 -> PING!
@mswelker Could you take a screenshot of the io.adafruit.com page for the units having the packet decode error?
The Huzzah ESP32 V2's with beta 67 (unit 1 and 7) are connecting to io.adafruit.com and I can control the neopixels remotely as well as scan and detect i2c addresses, but no data is recorded from the i2c sensors connected to it. Screenshots of the device page is attached. I had similar problems with a Huzzah ESP32-S2 w/ BME280 board (unit 8) when it was running beta67, but its currently working again after I reflashed it with beta66, so I'm including its device page screenshot as well.
@mswelker This should be resolved by adafruit/Adafruit_MQTT_Library#227 and will be included in WipperSnapper Beta 71 (released later today).
Please try beta 71 and let me know in this issue if the packet decode error has been resolved or is still persistent.