Frequent Core Panic: Exception was unhandled & Heap Poisoning
theelims opened this issue · comments
Describe the bug
emodbus keeps crashing my ESP32 with different Core Panics: "Exception was unhandled" and "Heap Poisoning" are the most frequent. This happens in particular when polling certain hold registers quite frequently (every 50ms).
I use syncRequest as all modbus transactions happen in their own FreeRTOS task and a blocking modbus access suits the flow in the code much better.
My use case is the polling of real time position and torque data from a servo motor over a RS232 modbus RTU connection. There are no other modbus devices connected.
Expected behavior
Stable performance, even when polling frequent data. No memory leaks.
Example code
// Initialisation
#include "ModbusClientRTU.h"
ModbusClientRTU _MB;
RTUutils::prepareHardwareSerial(Serial2);
Serial2.begin(57600, SERIAL_8E1, modbusRxPin, modbusTxPin);
_MB.setTimeout(2000);
_MB.begin(Serial2);
// modbus request running in a dedicated FreeRTOS task every 50ms
ModbusMessage response = _MB.syncRequest(token++, 1, READ_HOLD_REGISTER, address, 1);
Error err = response.getError();
if (err != SUCCESS)
{
ModbusError e(err);
ESP_LOGD("iHSVServoV6", "Error creating Modbus request at %#04x: %02X - %s", address, (int)e, (const char *)e);
}
else
{
uint16_t value;
int result;
response.get(3, value);
if (sign && value > 32676)
{
result = value - 65536;
}
else
{
result = value;
}
ESP_LOGV("iHSVServoV6", "Modbus read at %#04x: FC=%d, length=%d, value= %d", address, response.getFunctionCode(), response[2], result);
return result;
}To Reproduce
Let the syncRequest run in a task every 50ms. It will crash within a few minutes with one of the following core panics.
Backtrace
Heap Poisoning
CORRUPT HEAP: Bad tail at 0x3ffdf1b5. Expected 0xbaad5678 got 0xe0ad5678
assert failed: multi_heap_free multi_heap_poisoning.c:259 (head != NULL)
Backtrace: 0x40084945:0x3ffd8920 0x4008f515:0x3ffd8940 0x400951d1:0x3ffd8960 0x40094dc9:0x3ffd8a90 0x40084eb5:0x3ffd8ab0 0x40095201:0x3ffd8ad0 0x401854c1:0x3ffd8af0 0x400e4716:0x3ffd8b10 0x400e4761:0x3ffd8b30 0x400e51e2:0x3ffd8b50 0x400e4025:0x3ffd8b80 0x400de382:0x3ffd8bb0 0x400de41a:0x3ffd8c00 0x400de78f:0x3ffd8c60 0x400ddaae:0x3ffd8c90
#0 0x40084945:0x3ffd8920 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:408
#1 0x4008f515:0x3ffd8940 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:137
#2 0x400951d1:0x3ffd8960 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
#3 0x40094dc9:0x3ffd8a90 in multi_heap_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/multi_heap_poisoning.c:259 (discriminator 1)
#4 0x40084eb5:0x3ffd8ab0 in heap_caps_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/heap_caps.c:382
#5 0x40095201:0x3ffd8ad0 in free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/heap.c:39
#6 0x401854c1:0x3ffd8af0 in operator delete(void*) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49
#7 0x400e4716:0x3ffd8b10 in __gnu_cxx::new_allocator<unsigned char>::deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\ext/new_allocator.h:125
(inlined by) std::allocator_traits<std::allocator<unsigned char> >::deallocate(std::allocator<unsigned char>&, unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/alloc_traits.h:462
(inlined by) std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:304
#8 0x400e4761:0x3ffd8b30 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::~_Vector_base() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:285
(inlined by) std::vector<unsigned char, std::allocator<unsigned char> >::~vector() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:570
(inlined by) ModbusMessage::~ModbusMessage() at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusMessage.cpp:20
#9 0x400e51e2:0x3ffd8b50 in std::pair<unsigned int const, ModbusMessage>::~pair() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_pair.h:208
(inlined by) void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > >::destroy<std::pair<unsigned int const, ModbusMessage> >(std::pair<unsigned int const, ModbusMessage>*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\ext/new_allocator.h:140
(inlined by) void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > > >::destroy<std::pair<unsigned int const, ModbusMessage> >(std::allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > >&, std::pair<unsigned int const, ModbusMessage>*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/alloc_traits.h:487
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_destroy_node(std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> >*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:650
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_drop_node(std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> >*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:658
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:2500
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:1148
(inlined by) std::map<unsigned int, ModbusMessage, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_map.h:1037
(inlined by) ModbusClient::waitSync(unsigned char, unsigned char, unsigned int) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClient.cpp:95
#10 0x400e4025:0x3ffd8b80 in ModbusClientRTU::syncRequestM(ModbusMessage, unsigned int) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClientRTU.cpp:173 (discriminator 1)
#11 0x400de382:0x3ffd8bb0 in ModbusMessage ModbusClient::syncRequest<int, Modbus::FunctionCode, unsigned short&, int>(unsigned int, int&&, Modbus::FunctionCode&&, unsigned short&, int&&) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClient.h:54 (discriminator 1)
#12 0x400de41a:0x3ffd8c00 in iHSVServoV6Motor::_readServoRegister(unsigned short, bool) at lib/StrokeEngine/motor/ihsvServoV6.h:561
#13 0x400de78f:0x3ffd8c60 in iHSVServoV6Motor::_reportMotionPoint() at lib/StrokeEngine/motor/ihsvServoV6.h:808
#14 0x400ddaae:0x3ffd8c90 in MotorInterface::_positionFeedbackTask() at lib/StrokeEngine/motor/motor.h:340
(inlined by) MotorInterface::_positionFeedbackTaskImpl(void*) at lib/StrokeEngine/motor/motor.h:327
Exception was unhandled
Guru Meditation Error: Core 0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC : 0x400942d6 PS : 0x00060f33 A0 : 0x80094c9b A1 : 0x3ffd7160
A2 : 0x3ffcc3fc A3 : 0x0000003b A4 : 0xfffffffc A5 : 0xbaad5678
A6 : 0x00060d20 A7 : 0x00000001 A8 : 0xbaad5678 A9 : 0x00000000
A10 : 0x00000002 A11 : 0x0000003a A12 : 0x3ffd9100 A13 : 0x00060f23
A14 : 0x00060f23 A15 : 0x00000000 SAR : 0x0000001a EXCCAUSE: 0x0000001c
EXCVADDR: 0xbaad567c LBEG : 0x4008abc5 LEND : 0x4008abe7 LCOUNT : 0x00000000
Backtrace: 0x400942d3:0x3ffd7160 0x40094c98:0x3ffd7180 0x40094dd2:0x3ffd71a0 0x40084eb5:0x3ffd71c0 0x40095201:0x3ffd71e0 0x401854c1:0x3ffd7200 0x400e4716:0x3ffd7220 0x400e4761:0x3ffd7240 0x400e44f3:0x3ffd7260
#0 0x400942d3:0x3ffd7160 in block_merge_prev at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/heap_tlsf.c:343
(inlined by) tlsf_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/heap_tlsf.c:966
#1 0x40094c98:0x3ffd7180 in multi_heap_free_impl at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/multi_heap.c:212
#2 0x40094dd2:0x3ffd71a0 in multi_heap_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/multi_heap_poisoning.c:266
#3 0x40084eb5:0x3ffd71c0 in heap_caps_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/heap_caps.c:382
#4 0x40095201:0x3ffd71e0 in free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/heap.c:39
#5 0x401854c1:0x3ffd7200 in operator delete(void*) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49
#6 0x400e4716:0x3ffd7220 in __gnu_cxx::new_allocator<unsigned char>::deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\ext/new_allocator.h:125
(inlined by) std::allocator_traits<std::allocator<unsigned char> >::deallocate(std::allocator<unsigned char>&, unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/alloc_traits.h:462
(inlined by) std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:304
#7 0x400e4761:0x3ffd7240 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::~_Vector_base() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:285
(inlined by) std::vector<unsigned char, std::allocator<unsigned char> >::~vector() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:570
(inlined by) ModbusMessage::~ModbusMessage() at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusMessage.cpp:20
#8 0x400e44f3:0x3ffd7260 in ModbusClientRTU::RequestEntry::~RequestEntry() at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClientRTU.h:63 (discriminator 1)
(inlined by) ModbusClientRTU::handleConnection(ModbusClientRTU*) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClientRTU.cpp:246 (discriminator 1)```
Looks like you are blasting the task's stack or heap. You could try to give it a larger stack when the task is created than the 4kb used now and see if the crashes will happen less frequently.
50ms is a rather high polling frequency by the way.
Does eModbus have it's own stack size, or are you referring to the stack size of the task I'm creating?
eModbus is using xCreateTask to start the server or client background tasks. This call is made with the stack size for the new tadks.
How do I change that? Is there a build flag?
I am afraid you will have to alter the ModbusClientRTU.cpp file in the library sources to change the value. The 4k was sufficient in all applications so far, you are the first to report an issue.
I'll test that and report back.
Doubling to 8k doesn't have any positive impact. It keeps crashing with the aforementioned kernel panics.
Okay, but it was worth trying. Both the crashes you provided the logs are happening when trying to delete a syncRequest from an internal STL map, and both are occurring deep inside the STL code.
The only thing I can imagine here is a race condition where two tasks are attempting to do the same. Can you elaborate a bit on how you are doing the requests? What priority does your dedicated task have? It also would be helpful to have the eModbus log lines immediately before the crash with the verbose logging switched on (-DLOG_LEVEL=LOG_LEVEL_VERBOSE).
ModbusClientRTU is a class member. In that class the following task is created:
xTaskCreatePinnedToCore(
_positionFeedbackTaskImpl, // Function that should be called
"Motion Feedback", // Name of the task (for debugging)
4096, // Stack size (bytes)
this, // Pass reference to this class instance
10, // Pretty high task priority
&_taskPositionFeedbackHandle, // Task handle
1 // Pin to application core
);This task polls with vTaskDelayUntil every 50ms these two functions to read a 32-bit register and a 16-bit register:
/**************************************************************************/
/*!
@brief Read 16bit values from the servo register map. Retries up to 2x if
a read fails. Returns 0 if all retries failed.
@param address Address of the register to read.
@param sign Set to true if the register is signed.
@return value of the register.
*/
/**************************************************************************/
int _readServoRegister(uint16_t address, boolean sign = false)
{
for (size_t i = 0; i < 3; i++)
{
ModbusMessage response = _MB.syncRequest(token++, 1, READ_HOLD_REGISTER, address, 1);
Error err = response.getError();
if (err != SUCCESS)
{
ModbusError e(err);
ESP_LOGD("iHSVServoV6", "Error creating Modbus request at %#04x: %02X - %s", address, (int)e, (const char *)e);
}
else
{
uint16_t value;
int result;
response.get(3, value);
if (sign && value > 32676)
{
result = value - 65536;
}
else
{
result = value;
}
ESP_LOGV("iHSVServoV6", "Modbus read at %#04x: FC=%d, length=%d, value= %d", address, response.getFunctionCode(), response[2], result);
return result;
}
}
return 0;
}
/**************************************************************************/
/*!
@brief Read 32bit values from the servo register map. Retries up to 2x if
a read fails. Returns 0 if all retries failed.
@param address Address of the register to read.
@param sign Set to true if the register is signed.
@return value of the register.
*/
/**************************************************************************/
int _readServoRegister32bit(uint16_t address, boolean sign = false)
{
for (size_t i = 0; i < 3; i++)
{
ModbusMessage response = _MB.syncRequest(token++, 1, READ_HOLD_REGISTER, address, 2);
Error err = response.getError();
if (err != SUCCESS)
{
ModbusError e(err);
ESP_LOGE("iHSVServoV6", "Error creating Modbus request at %#04x: %02X - %s", address, (int)e, (const char *)e);
}
else
{
uint32_t value;
int result;
response.get(3, value);
if (sign && value > 2147482647)
{
result = value - 4294967296;
}
else
{
result = value;
}
ESP_LOGV("iHSVServoV6", "Modbus 32bit read at %#04x: FC=%d, length=%d, value=%d", address, response.getFunctionCode(), response[2], result);
return result;
}
}
return 0;
}I added the loop with 3 retries because I kept getting sporadic (every 2-3s) CRC errors (E2) and very rarely E5 on the 32-bit read operation.
Here is a full second log before a crash (which is also a mechanical crash, because it drives the servo violently into its mechanical end-stop):
[D] 156019| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156022| RTUutils.cpp [ 261] receive: C/1762us without data after 7
[V] 156033| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFD9098/7:
| 0000: 01 03 02 FF FF B9 F4 |....... |
[D] 156044| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD95D8/5:
| 0000: 01 03 02 FF FF |..... |
[D] 156055| ModbusClientRTU.cpp [ 268] handleConnection: Data response (5 bytes) received.
[V] Data: @3FFD95D8/5:
| 0000: 01 03 02 FF FF |..... |
[D] 156076| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD95D8/5:
| 0000: 01 03 02 FF FF |..... |
[V] 156096| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156096| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156097| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFDFD00/6:
| 0000: 01 03 08 36 00 02 |...6.. |
[D] 156119| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156131| RTUutils.cpp [ 261] receive: C/1755us without data after 9
[V] 156132| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFD9098/9:
| 0000: 01 03 04 FF FF FF CC BB B2 |......... |
[D] 156143| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFDFD00/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156164| ModbusClientRTU.cpp [ 268] handleConnection: Data response (7 bytes) received.
[V] Data: @3FFDFD00/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156174| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFDFD00/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[V] 156197| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156197| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156198| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD9020/6:
| 0000: 01 03 08 44 00 01 |...D.. |
[D] 156221| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156234| RTUutils.cpp [ 261] receive: C/1764us without data after 7
[V] 156235| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFD9098/7:
| 0000: 01 03 02 00 07 F9 86 |....... |
[D] 156256| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[D] 156266| ModbusClientRTU.cpp [ 268] handleConnection: Data response (5 bytes) received.
[V] Data: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[D] 156288| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[V] 156308| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156308| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156309| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD420C/6:
| 0000: 01 03 08 36 00 02 |...6.. |
[D] 156332| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156334| RTUutils.cpp [ 261] receive: C/1757us without data after 9
[V] 156345| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFD9098/9:
| 0000: 01 03 04 FF FF FF CC BB B2 |......... |
[D] 156356| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156377| ModbusClientRTU.cpp [ 268] handleConnection: Data response (7 bytes) received.
[V] Data: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156388| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[V] 156419| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156419| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156420| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD95B8/6:
| 0000: 01 03 08 44 00 01 |...D.. |
[D] 156445| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156447| RTUutils.cpp [ 261] receive: C/1759us without data after 7
[V] 156457| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFD9098/7:
| 0000: 01 03 02 00 07 F9 86 |....... |
[D] 156468| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[D] 156479| ModbusClientRTU.cpp [ 268] handleConnection: Data response (5 bytes) received.
[V] Data: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[D] 156500| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD95B8/5:
| 0000: 01 03 02 00 07 |..... |
[V] 156520| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156520| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156521| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD41B8/6:
| 0000: 01 03 08 36 00 02 |...6.. |
[D] 156543| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156556| RTUutils.cpp [ 261] receive: C/1755us without data after 9
[V] 156556| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFECE2C/9:
| 0000: 01 03 04 FF FF FF CC BB B2 |......... |
[D] 156567| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD41B8/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156588| ModbusClientRTU.cpp [ 268] handleConnection: Data response (7 bytes) received.
[V] Data: @3FFD41B8/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156599| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD41B8/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[V] 156621| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156621| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156622| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD9DA0/6:
| 0000: 01 03 08 44 00 01 |...D.. |
[D] 156647| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156660| RTUutils.cpp [ 261] receive: C/1765us without data after 7
[V] 156660| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFEF4BC/7:
| 0000: 01 03 02 FF FE 78 34 |.....x4 |
[D] 156681| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFDAC30/5:
| 0000: 01 03 02 FF FE |..... |
[D] 156692| ModbusClientRTU.cpp [ 268] handleConnection: Data response (5 bytes) received.
[V] Data: @3FFDAC30/5:
| 0000: 01 03 02 FF FE |..... |
[D] 156703| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFDAC30/5:
| 0000: 01 03 02 FF FE |..... |
[V] 156732| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156732| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156733| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD420C/6:
| 0000: 01 03 08 36 00 02 |...6.. |
[D] 156756| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156759| RTUutils.cpp [ 261] receive: C/1755us without data after 9
[V] 156770| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFECE2C/9:
| 0000: 01 03 04 FF FF FF CC BB B2 |......... |
[D] 156781| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156793| ModbusClientRTU.cpp [ 268] handleConnection: Data response (7 bytes) received.
[V] Data: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 156813| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[V] 156833| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156833| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156834| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD41B8/6:
| 0000: 01 03 08 44 00 01 |...D.. |
[D] 156857| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156860| RTUutils.cpp [ 261] receive: C/1758us without data after 7
[V] 156870| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFEF4BC/7:
| 0000: 01 03 02 FF FE 78 34 |.....x4 |
[D] 156881| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD41B8/5:
| 0000: 01 03 02 FF FE |..... |
[D] 156902| ModbusClientRTU.cpp [ 268] handleConnection: Data response (5 bytes) received.
[V] Data: @3FFD41B8/5:
| 0000: 01 03 02 FF FE |..... |
[D] 156913| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD41B8/5:
| 0000: 01 03 02 FF FE |..... |
[V] 156944| ModbusMessage.cpp [ 477] checkData: Check data #3
[D] 156944| ModbusClientRTU.cpp [ 230] addToQueue: RC=01
[D] 156945| ModbusClientRTU.cpp [ 248] handleConnection: Pulled request from queue
[D] Sent packet: @3FFD420C/6:
| 0000: 01 03 08 36 00 02 |...6.. |
[D] 156968| ModbusClientRTU.cpp [ 253] handleConnection: Request sent.
[V] 156971| RTUutils.cpp [ 261] receive: C/1753us without data after 9
[V] 156981| RTUutils.cpp [ 271] receive: C/[V] Raw buffer received: @3FFECE2C/9:
| 0000: 01 03 04 FF FF FF CC BB B2 |......... |
[D] 156992| RTUutils.cpp [ 436] receive: C/[D] Received packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 157004| ModbusClientRTU.cpp [ 268] handleConnection: Data response (7 bytes) received.
[V] Data: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
[D] 157027| ModbusClientRTU.cpp [ 289] handleConnection: Response generated.
[V] Response packet: @3FFD420C/7:
| 0000: 01 03 04 FF FF FF CC |....... |
CORRUPT HEAP: Bad tail at 0x3ffd9dc7. Expected 0xbaad5678 got 0xbaad7c78
assert failed: multi_heap_free multi_heap_poisoning.c:259 (head != NULL)
Backtrace: 0x40084945:0x3ffd8a30 0x4008f515:0x3ffd8a50 0x400951d1:0x3ffd8a70 0x40094dc9:0x3ffd8ba0 0x40084eb5:0x3ffd8bc0 0x40095201:0x3ffd8be0 0x40186c01:0x3ffd8c00 0x400e5b4a:0x3ffd8c20 0x400e5b95:0x3ffd8c40 0x400e692e:0x3ffd8c60 0x400e5339:0x3ffd8c90 0x400df3ae:0x3ffd8cc0 0x400df546:0x3ffd8d10 0x400df5fe:0x3ffd8d70 0x400ddc3a:0x3ffd8da0
#0 0x40084945:0x3ffd8a30 in panic_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/panic.c:408
#1 0x4008f515:0x3ffd8a50 in esp_system_abort at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/esp_system/esp_system.c:137
#2 0x400951d1:0x3ffd8a70 in __assert_func at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/assert.c:85
#3 0x40094dc9:0x3ffd8ba0 in multi_heap_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/multi_heap_poisoning.c:259 (discriminator 1)
#4 0x40084eb5:0x3ffd8bc0 in heap_caps_free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/heap/heap_caps.c:382
#5 0x40095201:0x3ffd8be0 in free at /Users/ficeto/Desktop/ESP32/ESP32S2/esp-idf-public/components/newlib/heap.c:39
#6 0x40186c01:0x3ffd8c00 in operator delete(void*) at /builds/idf/crosstool-NG/.build/HOST-x86_64-w64-mingw32/xtensa-esp32-elf/src/gcc/libstdc++-v3/libsupc++/del_op.cc:49
#7 0x400e5b4a:0x3ffd8c20 in __gnu_cxx::new_allocator<unsigned char>::deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\ext/new_allocator.h:125
(inlined by) std::allocator_traits<std::allocator<unsigned char> >::deallocate(std::allocator<unsigned char>&, unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/alloc_traits.h:462
(inlined by) std::_Vector_base<unsigned char, std::allocator<unsigned char> >::_M_deallocate(unsigned char*, unsigned int) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:304
#8 0x400e5b95:0x3ffd8c40 in std::_Vector_base<unsigned char, std::allocator<unsigned char> >::~_Vector_base() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:285
(inlined by) std::vector<unsigned char, std::allocator<unsigned char> >::~vector() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_vector.h:570
(inlined by) ModbusMessage::~ModbusMessage() at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusMessage.cpp:20
#9 0x400e692e:0x3ffd8c60 in std::pair<unsigned int const, ModbusMessage>::~pair() at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_pair.h:208
(inlined by) void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > >::destroy<std::pair<unsigned int const, ModbusMessage> >(std::pair<unsigned int const, ModbusMessage>*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\ext/new_allocator.h:140
(inlined by) void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > > >::destroy<std::pair<unsigned int const, ModbusMessage> >(std::allocator<std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> > >&, std::pair<unsigned int const, ModbusMessage>*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/alloc_traits.h:487
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_destroy_node(std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> >*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:650
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_drop_node(std::_Rb_tree_node<std::pair<unsigned int const, ModbusMessage> >*) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:658
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:2500
(inlined by) std::_Rb_tree<unsigned int, std::pair<unsigned int const, ModbusMessage>, std::_Select1st<std::pair<unsigned int const, ModbusMessage> >, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_tree.h:1148
(inlined by) std::map<unsigned int, ModbusMessage, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, ModbusMessage> > >::erase[abi:cxx11](std::_Rb_tree_iterator<std::pair<unsigned int const, ModbusMessage> >) at c:\users\elims\.platformio\packages\toolchain-xtensa-esp32\xtensa-esp32-elf\include\c++\8.4.0\bits/stl_map.h:1037
(inlined by) ModbusClient::waitSync(unsigned char, unsigned char, unsigned int) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClient.cpp:95
#10 0x400e5339:0x3ffd8c90 in ModbusClientRTU::syncRequestM(ModbusMessage, unsigned int) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClientRTU.cpp:173 (discriminator 1)
#11 0x400df3ae:0x3ffd8cc0 in ModbusMessage ModbusClient::syncRequest<int, Modbus::FunctionCode, unsigned short&, int>(unsigned int, int&&, Modbus::FunctionCode&&, unsigned short&, int&&) at .pio/libdeps/OSSMRefBoard/eModbus/src/ModbusClient.h:54 (discriminator 1)
#12 0x400df546:0x3ffd8d10 in iHSVServoV6Motor::_readServoRegister32bit(unsigned short, bool) at lib/StrokeEngine/motor/ihsvServoV6.h:558
#13 0x400df5fe:0x3ffd8d70 in iHSVServoV6Motor::_reportMotionPoint() at lib/StrokeEngine/motor/ihsvServoV6.h:767
#14 0x400ddc3a:0x3ffd8da0 in MotorInterface::_positionFeedbackTask() at lib/StrokeEngine/motor/motor.h:371
(inlined by) MotorInterface::_positionFeedbackTaskImpl(void*) at lib/StrokeEngine/motor/motor.h:358
Thanks. This is pointing roughly again to a corrupted heap, but with a different twist. While the first two crashes did happen when attempting to delete the obsolete map entry with the sync request after the request had been finished, this now is at the point where a local ModbusMessage is to be deleted upon leaving the syncRequest call.
From what I have seen so far both should be safe. The various crash reasons all have in common that the addressed heap area seems already to have been overwritten at the time of the crashing calls.
I would reckon the reason is another task doing the same (so the area had been freed already) or interfering time-wise. The latter should not be possible with RTOS normally.
- If you will double your polling interval to 100ms - do the crashes happen less frequently?
- Can you print out the free heap before and after the
syncRequestcalls? - If the read register addresses are not too far apart and adresses in between are valid, it could be advantageous to request the whole section as one and pick both values from it. That way you would half the request overhead.
BTW: both E2 and E5 errors will happen if there are timing issues on the bus - adapter delays or servers driven to their processing limits.
If I double the polling interval the crashes happen after a similar time.
However, if the motor is not running and the step generation for it is not active it runs for hours instead of a few minutes before crashing. There might be a processing bottle neck involved as well.
The heap values I need to provide at a later time.
That (double polling time, same outcome) is pointing to some other task colliding with the Modbus communications.
Seems to have found a resolution.