Spoa-mirror crash when http get request
AntoninGarnaud opened this issue · comments
Hello,
I'm trying to do the tutorial regarding traffic mirroring (https://www.haproxy.com/fr/blog/haproxy-traffic-mirroring-for-real-world-testing/)
Unfortunately i have a core dump generated during the first http get request.
I'm using centos 7, HA-Proxy version 2.0.1-1.0.0-219.645 2020/03/27 and spoa-mirror v1.2.9
spoa-mirror v1.2.9 [build 2407] by Miroslav Zagorac <mzagorac@haproxy.com>, Apr 15 2020
Here is what gdb shows with a bt and a full bt.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000405213 in mir_ptr_free (data=0x7fa0294abc20) at spoa-message.c:499
499 list_for_each_entry_safe(hdr, hdr_back, (*data)->hdrs, list)
(gdb) bt
#0 0x0000000000405213 in mir_ptr_free (data=0x7fa0294abc20) at spoa-message.c:499
#1 0x0000000000404fad in spoa_msg_mirror (frame=0x7fa02400f430, buf=0x7fa0294abcc0, end=0x7fa02400f670 "") at spoa-message.c:456
#2 0x0000000000406351 in process_frame_cb (loop=0x7fa0240008c0, ev=0x7fa02400f468, revents=256) at spoa.c:348
#3 0x00007fa02ddff587 in ev_invoke_pending () from /lib64/libev.so.4
#4 0x00007fa02de0278d in ev_run () from /lib64/libev.so.4
#5 0x0000000000411d2d in worker_thread (data=0x1af5d40) at worker.c:300
#6 0x00007fa02dbe6e65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fa02d90f88d in clone () from /lib64/libc.so.6
(gdb) bt full
#0 0x0000000000405213 in mir_ptr_free (data=0x7fa0294abc20) at spoa-message.c:499
hdr = 0x0
hdr_back = 0x7fa0240135c0
__func__ = "mir_ptr_free"
#1 0x0000000000404fad in spoa_msg_mirror (frame=0x7fa02400f430, buf=0x7fa0294abcc0, end=0x7fa02400f670 "") at spoa-message.c:456
data = {boolean = 57, int32 = 604042553, uint32 = 604042553, int64 = 140325775537465, uint64 = 140325775537465, ipv4 = {s_addr = 604042553}, ipv6 = {__in6_u = {
__u6_addr8 = "9\365\000$\240\177\000\000,\001\000\000\000\000\000", __u6_addr16 = {62777, 9216, 32672, 0, 300, 0, 0, 0}, __u6_addr32 = {604042553, 32672, 300, 0}}}, chk = {
ptr = 0x7fa02400f539 "\004host\ntest.lba.fr\nuser-agentNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0\006acceptJtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\017accept-lang"..., len = 300}}
type = SPOE_DATA_T_BIN
mir = 0x7fa024013500
ptr = 0x7fa02400f665 "\barg_body\t"
str = 0x7fa02400f52e "arg_hdrs\t\374\003\004host\ntest.lba.fr\nuser-agentNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0\006acceptJtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\017"...
len = 8
nbargs = 5 '\005'
i = 4
retval = -1
__func__ = "spoa_msg_mirror"
#2 0x0000000000406351 in process_frame_cb (loop=0x7fa0240008c0, ev=0x7fa02400f468, revents=256) at spoa.c:348
frame = 0x7fa02400f430
ptr = 0x7fa02400f4fa "\005\narg_method\b\003GET\barg_path\b\n/alti/test\aarg_ver\b\003\061.1\barg_hdrs\t\374\003\004host\ntest.lba.fr\nuser-agentNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0\006acceptJtext/html,application/x"...
str = 0x7fa02400f4f4 "mirror\005\narg_method\b\003GET\barg_path\b\n/alti/test\aarg_ver\b\003\061.1\barg_hdrs\t\374\003\004host\ntest.lba.fr\nuser-agentNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0\006acceptJtext/html,applica"...
end = 0x7fa02400f670 ""
buf = 0x0
len = 6
rc = 7
ip_score = -42
__func__ = "process_frame_cb"
#3 0x00007fa02ddff587 in ev_invoke_pending () from /lib64/libev.so.4
No symbol table info available.
#4 0x00007fa02de0278d in ev_run () from /lib64/libev.so.4
No symbol table info available.
#5 0x0000000000411d2d in worker_thread (data=0x1af5d40) at worker.c:300
name = "sm/wrk: 1\000\000\000\000\000\000"
c = 0x7fa02e48c000
cback = 0x7fa0294b2700
f = 0x0
fback = 0x0
w = 0x1af5d40
__func__ = "worker_thread"
#6 0x00007fa02dbe6e65 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00007fa02d90f88d in clone () from /lib64/libc.so.6
Here is also the spoa-mirror log file
[ 0][ 0.001278] --- start --- 2020-04-19 23:03:30 -------------------------
[ 1][ 5.945418] <1:29> --> Receiving data
[ 1][ 5.945462] <1:29> 4/4/4 byte(s) received frame length
[ 1][ 5.945481] <1:29> 129/129/129 byte(s) received frame data
[ 1][ 5.945491] <1:29> New frame of 129 bytes received: <0100000001000012737570706f727465642d76657273696f6e730803322e300e6d61782d6672616d652d73697a6503fcf0060c6361706162696c69746965730810706970656c696e696e672c6173796e6309656e67696e652d6964082463306633393765662d373930392d343066362d613238372d326532663239636435323838> <........supported-versions..2.0.max-frame-size.....capabilities..pipelining,async.engine-id.$c0f397ef-7909-40f6-a287-2e2f29cd5288>
[ 1][ 5.945503] <1> --> HAPROXY-HELLO decoding frame
[ 1][ 5.945519] <1> --> HAPROXY-HELLO header 1 0x01000000 0 0
[ 1][ 5.945525] <1> --> HAPROXY-HELLO stream-id=0 - frame-id=0
[ 1][ 5.945533] <1> K/V item: key=supported-versions
[ 1][ 5.945552] <1> --> HAPROXY-HELLO supported versions: 2.0
[ 1][ 5.945555] <1> K/V item: key=max-frame-size
[ 1][ 5.945559] <1> --> HAPROXY-HELLO maximum frame size: 16380
[ 1][ 5.945561] <1> K/V item: key=capabilities
[ 1][ 5.945563] <1> --> HAPROXY-HELLO capabilities check: pipelining,async
[ 1][ 5.945568] <1> --> HAPROXY-HELLO capabilities set: pipelining async
[ 1][ 5.945571] <1> K/V item: key=engine-id
[ 1][ 5.945577] <1> --> HAPROXY-HELLO engine id: c0f397ef-7909-40f6-a287-2e2f29cd5288
[ 1][ 5.945585] <1> <-- AGENT-HELLO encoding frame
[ 1][ 5.945597] <1> <-- AGENT-HELLO stream-id=0 - frame-id=0
[ 1][ 5.945599] <1> <-- AGENT-HELLO version: 2.0
[ 1][ 5.945601] <1> <-- AGENT-HELLO maximum frame size: 16380
[ 1][ 5.945602] <1> <-- AGENT-HELLO capabilities:
[ 1][ 5.945614] <1:29> <-- Sending data
[ 1][ 5.945697] <1:29> 4/4/4 byte(s) send frame length
[ 1][ 5.945724] <1:29> 54/54/54 byte(s) send frame data
[ 1][ 5.945729] <1:29> Frame of 54 bytes sent: <650000000100000776657273696f6e0803322e300e6d61782d6672616d652d73697a6503fcf0060c6361706162696c69746965730800> <e.......version..2.0.max-frame-size.....capabilities..>
[ 1][ 5.945882] <1:29> --> Receiving data
[ 1][ 5.945888] <1:29> 4/4/4 byte(s) received frame length
[ 1][ 5.945893] <1:29> 388/388/388 byte(s) received frame data
[ 1][ 5.945906] <1:29> New frame of 388 bytes received: <03000000010001066d6972726f72050a6172675f6d6574686f640803474554086172675f70617468080a2f616c74692f74657374076172675f7665720803312e31086172675f6864727309fc0304686f73740a7778732e69676e2e66720a757365722d6167656e744e4d6f7a696c6c612f352e30202857696e646f7773204e542031302e303b2057696e36343b207836343b2072763a37352e3029204765636b6f2f32303130303130312046697265666f782f37352e30066163636570744a746578742f68746d6c2c6170706c69636174696f6e2f7868746d6c2b786d6c2c6170706c69636174696f6e2f786d6c3b713d302e392c696d6167652f776562702c2a2f2a3b713d302e380f6163636570742d6c616e677561676523656e2d55532c656e3b713d302e382c66723b713d302e352c66722d46523b713d302e330f6163636570742d656e636f64696e670d677a69702c206465666c61746519757067726164652d696e7365637572652d726571756573747301310000086172675f626f64790900> <........mirror..arg_method..GET.arg_path../alti/test.arg_ver..1.1.arg_hdrs....host.test.lba.fr.user-agentNMozilla/5.0 (Windows NT 10.0; Win64; x64; rv:75.0) Gecko/20100101 Firefox/75.0.acceptJtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8.accept-language#en-US,en;q=0.8,fr;q=0.5,fr-FR;q=0.3.accept-encoding.gzip, deflate.upgrade-insecure-requests.1...arg_body..>
[ 1][ 5.945911] <1> --> HAPROXY-NOTIFY decoding frame
[ 1][ 5.945913] <1> --> HAPROXY-NOTIFY header 3 0x01000000 0 1
[ 1][ 5.945916] <1> --> HAPROXY-NOTIFY stream-id=0 - frame-id=1
[ 1][ 5.945918] <1> --> HAPROXY-NOTIFY - unfragmented frame received - frag_len=0 - len=388 - offset=7
[ 1][ 5.945928] <1> Process frame messages: stream-id=0 - frame-id=1 - length=381 bytes
[ 1][ 5.945931] <1> Process SPOE Message 'mirror'
[ 1][ 5.945939] <1> 5 arg(s) expected
And the output of haproxy log
SPOE: [mirror] <EVENT:on-frontend-http-request> sid=0 st=257 0/0/16/-1/17 0/0 0/0 1/1
Thank you for your reply
Hello @AntoninGarnaud,
I apologize for not answering so far. I should have some free time soon so I'll pay attention to this.
Hello @AntoninGarnaud,
please check if the latest version of the program solves your problem.
Best regards,
Miroslav Zagorac
Hi,
I see your last commit about http header
I will try the 1.2.13 release.
Thank you,
Antonin garnaud
Hello, @zaga00
So I try your latest release (1.2.13) and it fix the core dump.
It's good news for us.
Thank you
Hello @AntoninGarnaud,
thank you for reporting the bug and for your patience.