Streaming using code built with PlatformIO fails rapidly with: Stream crash - ESP_ERR_HTTPD_RESP_SEND

Question

Streaming using code built with PlatformIO fails rapidly with: Stream crash - ESP_ERR_HTTPD_RESP_SEND

TheWolf95 opened this issue 2 years ago · comments

Hello!

I'm having issues having a reliable stream, since in short times (random, but between 2 to 30 seconds) the stream crashes, ending itself due to ESP_ERR_HTTPD_RESP_SEND. Which, looking to the ESP-IDF wiki, is due to sending an invalid chunk packet, confirmed by Chrome JS console returning ERR_INVALID_CHUNK_ENCODING 200 (OK).
I've been searching around for a solution, but I couldn't make it far. I also tried forcing ESP ignore the invalid packet error changing res into ESP_OK, but I guess once it happens Chrome just closes that stream connection, returning the error from the first if too.

I do not know if this is of any help, but it happens always on the third if in the stream handler method.

Thank you for your amazing work so far and in advance for your assistance!

Owen Carter · Answer 1 · Sun Mar 06 2022 00:54:00 GMT+0800 (China Standard Time)

Thankyou for the report;

Each frame of the stream is returned in response to a request sent by the browser (that's how MJPEG streams work)
ESP_ERR_HTTPD_RESP_SEND indicates that sending the response failed; that's all the cam knows. Since it has seen a failure it will close the stream, stopping it from doing that will fail anyway because:
The ERR_INVALID_CHUNKED_ENCODING (I assume you mis-typed above) is chrome saying 'i got a invalid response', it will then stop requesting new frames and the stream ends anyway.

It could be a esp bug that causes an invalid frame/chunk to be sent, but nothing showed up specifically on google; and I do not see this happening on my home cameras so I suspect it is something in the network causing regular packet loss.

I'll keep an eye out for it though, again, thanks for the report.

Owen Carter · Answer 2 · Fri Mar 11 2022 23:18:22 GMT+0800 (China Standard Time)

And... I'm seeing this now. Interesting. Think it is still something network, rather than code related.
Will monitor and reproduce.

Owen Carter · Answer 3 · Sat Mar 12 2022 01:06:14 GMT+0800 (China Standard Time)

Question: Are you using PlatformIO?

I can reproduce this when I commandline build using platformio; but when I build the same code using the ArduinoIDE the problem disappears.

Dawid Karoński · Answer 4 · Mon Mar 14 2022 06:37:26 GMT+0800 (China Standard Time)

Same issue here when using PlatformIO. The stream frezzes (fails) after a few seconds. I tried with Arduino IDE but I couldn't make the stream run with Arduino IDE at all because of SPIFFS: mount failed, -10025 and error page...

Owen Carter · Answer 5 · Mon Mar 14 2022 18:54:47 GMT+0800 (China Standard Time)

@Bulduper : With the IDE; double check the partition scheme you selected has SPIFFS and/or lower the x-clock rate. Also make sure you are using the HEAD revision RC1, not one of the older betas (which are more susceptible to this). This is discussed further in the discussion threads.

Owen Carter · Answer 6 · Mon Mar 14 2022 19:12:14 GMT+0800 (China Standard Time)

I have spent some time testing this: I see no obvious fix and I am not willing to spend more time on it; platformio is a nice-to-have, and not the focus of this work
If others want to investigate this further I will be happy to integrate a fix later.

Working; Arduino IDE with ESP core 2.0.2:

Firmware: Mar 14 2022 @ 10:48:09 (base: 4.0.rc1)
Sketch Size: 1004944 (total: 1966080, 51.0% used)
ESP sdk: v4.4-beta1-189-ga79dc75f0a

Failing: Platformio (current config)

platform = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
platform_packages = framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git#2.0.2
board = esp32dev
board_build.partitions = min_spiffs.csv
framework = arduino
build_flags =
    -DBOARD_HAS_PSRAM
    -mfix-esp32-psram-cache-issue

Firmware: Mar 14 2022 @ 10:23:19 (base: 4.0.rc1)
Sketch Size: 1020544 (total: 1966080, 51.0% used)
ESP sdk: v4.4-beta1-189-ga79dc75f0a

I note that the Platformio build with (nominally) the same SDK is 16k or so bigger in size. so the two builds are not equivalent despite the SDK giving the same version.

I tried a number of ways of configuring the platform, platform_packages and framework fields in platform.ini to no avail, either the build would fail or the error would persist. Using the latest (master) tag for the arduino-esp32 package seemed to make the problem worse. I also tried other frame sizes, disabling OTA, using a different AP and browser. Nothing worked.

https://github.com/search?q=ESP_ERR_HTTPD_RESP_SEND&type=issues shows that this is a pretty unique error.
Googling ESP_ERR_HTTPD_RESP_SEND isn't much help either.

As noted above; I will not close this; but I do not have any more time to work on it.
I'll document that PlatformIO builds are broken for the 4.0 release; but for the present I wont go any further.
If anybody wants to pursue this as a PlatformIO bug/question please go ahead; I'll assist where I can and hotfix any resolution.

TheWolf95 · Answer 7 · Tue Mar 15 2022 22:02:31 GMT+0800 (China Standard Time)

Hi!

Sorry for not answering before. I can confirm I was using PlatformIO to build and flash the code on ESP32-CAM and it wouldn't work.
Some time later I tried using Arduino IDE, as the included example was working fine (the simple one included with ESP libraries for Arduino), and it happened to work just fine. No more crashes and no more error, so the issue is somehow strictly due to PIO.

@Bulduper, I had the same issue too. You could simply disable custom config options by uncommenting #define NO_FS around line 165 of myconfig.h. Instead, if you want to use this code, for now, you have to use Arduino IDE and select under Tools > Partition scheme > Default 4MB or Minimal. This does exclude OTA, but works for me.
Also by commenting with ; before board_build.partitions = min_spiffs.csv in platformIO config does the job too, but stream crashes due to other issues.

@easytarget thank you for taking the time to investigate, as for now using ArduinoIDE is not an issue for me, so I guess we will just wait for PIO to fix their issues

Dawid Karoński · Answer 8 · Wed Mar 16 2022 01:56:04 GMT+0800 (China Standard Time)

Thank you @TheWolf95 ! Indeed your solution for IDE worked.

Florian · Answer 9 · Mon Mar 21 2022 03:44:19 GMT+0800 (China Standard Time)

Hello!

I have a working version built with platformio (v4.0 release).
The only things I've changed are the following:

in platformio.ini, i simply use platform = espressif32 and commented out platform_packages
in the main file i commented out line 383 config.grab_mode = CAMERA_GRAB_LATEST; since i got an "... is not defined" error

This program is up and running for several hours now and no problems.
How can i contribute to solve this issue?
Greetings

Andrew Black · Answer 10 · Tue Mar 22 2022 04:51:29 GMT+0800 (China Standard Time)

Thank you @mitsubishievo99 - I can confirm that making those changes worked for me, and that building with PlatformIO now results in a working program.

marspd · Answer 11 · Fri Mar 25 2022 22:26:14 GMT+0800 (China Standard Time)

Thank you @mitsubishievo99 - I can ALSO confirm that making those changes worked for me, and that building with PlatformIO now results in a working program.

Owen Carter · Answer 12 · Fri Mar 25 2022 23:58:05 GMT+0800 (China Standard Time)

@mitsubishievo99 @TheHoodedFoot @marspd
Yes; this is very valid as a workaround for the time being. But be warned that it may break in the future.

What you are doing is relying on the fact that the PlatformIO espressif32 package provides a older version of the Ardionio Core (V2.0.0, I believe, maybe 2.0.1, or some commit between the two).
This earlier Arduino Core has an older camera driver which doesnt support the .grab_mode, so you have to comment that out too.
If and when the platformio esp32 package is updated to the latest core this will break again, but (what with one thing and another) I'm not expecting this to happen any time soon. I'm actually investigating the root issue separately, so I hope to have a working platformio build eventually.

llamaonaskateboard · Answer 13 · Sun Apr 24 2022 18:06:29 GMT+0800 (China Standard Time)

The latest git version of framework-arduinoespressif32 (appears to be a RC of 2.0.3) has mysteriously fixed this issue for me.

platform = https://github.com/platformio/platform-espressif32.git#feature/arduino-upstream
platform_packages = framework-arduinoespressif32@https://github.com/espressif/arduino-esp32.git

PLATFORM: Espressif 32 (3.3.1+sha.3784198) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 2.0.3+sha.b3c203d
 - tool-esptoolpy 1.30100.210531 (3.1.0)
 - tool-mkspiffs 2.230.0 (2.30)
 - toolchain-xtensa-esp32 8.4.0+2021r2-patch3

Owen Carter · Answer 14 · Thu May 05 2022 23:13:49 GMT+0800 (China Standard Time)

Curious; thankyou for the update @llamaonaskateboard , I will re-test when I can, it would be nice if this has been a transient issue.

Owen Carter · Answer 15 · Sun May 08 2022 14:55:48 GMT+0800 (China Standard Time)

Ok, so with the latest Arduino core (2.0.3) specified the problem is resolved.

Streams are now stable for hours

Thank you again to all the folks who contributed and assisted with this.