Unexpected error: pool_crashed
wojciech-wilk opened this issue · comments
Hi,
I am doing MQTT load test. I was trying to connect 1M clients on 40 nodes with 8 vCPU each, but the poll has crashed multiple times with errors like cpu_sup:util() failed with reason timeout or cpu_sup:avg1() failed with reason timeout. It is correlated with a load peak on that server which reaches 10 when the typical load is ~2.5.
I would be grateful for any help
Hi Galhad,
I haven't seen mzbench pool crashing with this reason, are you testing VerneMQ or something different? What are MZBench/Erlang versions?
Thank you for a quick answer. I am testing VerneMQ, versions:
- VerneMQ 1.7.0
- MZBench 0.5.5
- Erlang 20
Both MZBench director and nodes are running as Docker containers.
My scenario is a single pool with worker_start = linear(5000 rpm)), I was trying different rate values, lower values seems to only delay the problem. I've attached screenshots where a high CPU is correlated with a peak in started workers per second. Please note that graphs show multiple failed benches (peaks). So it seems that for some reason worker_start rate peaks causing the CPU overload and results in the crash.
Bench system logs:
Worker metrics:
MQTT metrics:
CPU usage (peak):
Thanks for thorough answer,
MZBench 0.5.5 is old, there is a fork:
https://github.com/mzbench/mzbench
which supports newer OTP versions, it may be some compatibility errors,
please try and let me know if it works