flamegraph does not contain sys_enter_* calls with params
SergejIsbrecht opened this issue · comments
Version
Inferno 0.11.18
Issue
When using Inferno to process sampled perf script
output, some stacktraces a missing in a generated flamegraph.svg
perf record -F499 -e 'syscalls:sys_enter_*' --call-graph dwarf -- command
perf script > out
The generated flamegraph only contains sys calls without any parameters. For example:
File: out
XXXXXX XXXXX [005] 44427.957866: syscalls:sys_enter_sched_yield:
xxxxxxxxxx __sched_yield (/gpl/lib/libc)
...
xxxxxxxxxx [unknown] (/gpl/lib/libpthread)
xxxxxxxxxx [unknown] (/gpl/lib/libc)
XXXX XXXX [005] 44422.848908: syscalls:sys_enter_getuid:
XXXXXXXXXX getuid (/gpl/lib/libc)
...
XXXXXXXXXX __libc_start_main (/gpl/lib/libc)
XXXXXXXXXX _start (/opt/hmi/native_image/command)
This calls are visible in the flamegraph. On the contrary following entries are missing in the flamegraph.
XXXXXXXXX XXXXX [001] 44422.868561: syscalls:sys_enter_munmap: addr: 0xXXXXX, len: 0xXXXXX
XXXXXXX __munmap (/gpl/lib/libc)
....
XXXXXXX [unknown] (/gpl/lib/libpthread)
XXXXXXX [unknown] (/gpl/lib/libc)
Output: inferno-collapse-perf
perf script
XXXXXXXXXXXX XXXXX [003] 44422.850672: syscalls:sys_enter_sched_getaffinity: pid: 0xXXXXXXXXXX, len: 0xXXXXXXXXXX, user_mask_ptr: 0xXXXXXXXXXX
XXXXXXXXXX pthread_getaffinity_np (/gpl/lib/libpthread)
XXXXXXXXXX pthread_getattr_np (/gpl/lib/libpthread)
....
XXXXXXXXXX [unknown] (/gpl/lib/libpthread)
XXXXXXXXXX [unknown] (/gpl/lib/libc)
inferno-collapse-perf
[WARN inferno::collapse::perf::logging] Weird stack line: pid: 0xXXXXXXXXXX, len: 0xXXXXXXXXXX, user_mask_ptr: 0xXXXXXXXXXX
[WARN inferno::collapse::perf::logging] Weird event line: XXXXXXXXXX pthread_getaffinity_np (/gpl/lib/libpthread)
[WARN inferno::collapse::perf::logging] Weird event line: XXXXXXXXXX pthread_getattr_np (/gpl/lib/libpthread)
...
[WARN inferno::collapse::perf::logging] Weird event line: XXXXXXXXXX [unknown] (/gpl/lib/libpthread)
[WARN inferno::collapse::perf::logging] Weird event line: XXXXXXXXXX [unknown] (/gpl/lib/libc)
Perf
perf version 4.4
FlameGraph
Seems like FlameGraph by Brendan Gregg is able to pars perf script
properly
./stackcollapse-perf.pl
./flamegraph.pl
.
Example
perf 19953 [001] 33076.615333: syscalls:sys_enter_ioctl: fd: 0x0000040e, cmd: 0x00002400, arg: 0x00000000
112d8f [unknown] (/usr/lib/x86_64-linux-gnu/libc.so.6)
30bc56 __evlist__enable+0x96 (inlined)
26c177 __cmd_record+0x24b7 (inlined)
ffffffffffffffff [unknown] ([unknown])
pipewire-pulse 2110 [000] 33076.615351: syscalls:sys_enter_recvfrom: fd: 0x0000001a, ubuf: 0x56449a86f51c, size: 0x00000014, flags: 0x00000040, addr: 0x00000000, addr_len: 0x00000000
12063c __libc_recv+0x1ec (inlined)
12063c __libc_recv+0x1ec (inlined)
pipewire-pulse 2110 [000] 33076.615356: syscalls:sys_enter_epoll_wait: epfd: 0x00000004, events: 0x7ffd9af1c170, maxevents: 0x00000020, timeout: 0xffffffff
11eb66 epoll_wait+0x106 (inlined)
8a22 [unknown] (/usr/lib/x86_64-linux-gnu/spa-0.2/support/libspa-support.so)
56449a7bb8df [unknown] ([heap])
Oh interesting! This suggests we have a corner-case to handle in
Lines 377 to 421 in 86131d4
I suspect it's actually not the arguments that are tripping us up here, but rather the :
in the event name (syscalls:sys_enter_ioctl
). event
ends up getting set to syscalls
here
Lines 377 to 380 in 86131d4
and then the remainder (sys_enter_epoll
and the arguments) end up in post_event
here
Lines 396 to 408 in 86131d4
I'm somewhat short on time these days, so waiting for me to take a look at fixing this would probably take some time, but if you want to take a stab at adding a test case, tracing through the code here to figure out where it gets confused, and then submit a PR with a fix, I'd be happy to try and help guide you through!
I will give it a go, but I write Java most of the time. I have already been looking into the code base, but reading Rust is quite hard to me.
Anyhow, I think it should be easy to fix it after one is familiar with the code base.