3.2.0: build fails with `libbpf: failed to find BTF info for object procdump_ebpf.o`

Question

3.2.0: build fails with `libbpf: failed to find BTF info for object procdump_ebpf.o`

kloczek opened this issue 6 months ago · comments

cmake settings

-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
PROCDUMP_DISABLE_SYSTEM_LIBBPF:BOOL=OFF

cmake params and output:

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S .
-- The C compiler identification is GNU 14.0.1
-- The CXX compiler identification is GNU 14.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found ZLIB: /usr/lib64/libz.so (found version "1.3.0")
-- Found Libelf: /usr/lib64/libelf.so (found version "0.191")
-- Found Bpf:  (found version "1.4.0") found components: libbpf
-- Found Bpf:  (found version "1.4.0") found components: bpftool
-- Configuring done (1.5s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_Fortran_FLAGS_RELEASE
    INCLUDE_INSTALL_DIR
    LIB_INSTALL_DIR
    LIB_SUFFIX
    SHARE_INSTALL_PREFIX
    SYSCONF_INSTALL_DIR

And build fails with:

[ 24%] Built target ProcDumpProfiler
make  -f CMakeFiles/procdump_ebpf.dir/build.make CMakeFiles/procdump_ebpf.dir/depend
make[2]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
cd /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu/CMakeFiles/procdump_ebpf.dir/DependInfo.cmake "--color="
make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make  -f CMakeFiles/procdump_ebpf.dir/build.make CMakeFiles/procdump_ebpf.dir/build
make[2]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
[ 27%] Generating procdump.ebpf.o
/usr/sbin/bpftool gen object procdump.ebpf.o procdump_ebpf.o
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)
make[2]: *** [CMakeFiles/procdump_ebpf.dir/build.make:78: procdump.ebpf.o] Error 234
make[2]: *** Deleting file 'procdump.ebpf.o'
make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make[1]: *** [CMakeFiles/Makefile2:308: CMakeFiles/procdump_ebpf.dir/all] Error 2
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make: *** [Makefile:94: all] Error 2

Mario Hewardt · Answer 1 · Fri Apr 19 2024 23:17:47 GMT+0800 (China Standard Time)

Hi - Thanks for reporting this issue. What distribution is this and does it have BTF enabled (check - /sys/kernel/btf/vmlinux)?

Tomasz Kłoczko · Answer 2 · Fri Apr 19 2024 23:24:02 GMT+0800 (China Standard Time)

I'm using my own distribution but on build system is Fedora kernel.

[tkloczko@pers-jacek SPECS]$ ls -la /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 5923179 Apr 19 15:20 /sys/kernel/btf/vmlinux

Does it mean that what bpftool is doing depends on system running kernel? 🤔

Mario Hewardt · Answer 3 · Fri Apr 19 2024 23:47:34 GMT+0800 (China Standard Time)

The -restrack switch (memory tracking) is built using eBPF which utilizes BTF. Having said that, your system seems to be BTF enabled so there is something else going on. We build and produce packages for RHEL so there must be something specific to your distribution.

Try adding --debug to the bpf build steps in the cmake file. Towards the bottom you should see:

bpftool gen object procdump.ebpf.o procdump_ebpf.o

Change it to:

bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o

This may shed some more light on the problem.

Tomasz Kłoczko · Answer 4 · Sat Apr 20 2024 02:39:40 GMT+0800 (China Standard Time)

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o; echo

libbpf: linker: adding object file 'procdump_ebpf.o'...
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ ls -la procdump_ebpf.o
-rw-r--r-- 1 tkloczko tkloczko 27104 Apr 19 15:33 procdump_ebpf.o

Tomasz Kłoczko · Answer 5 · Sat Apr 20 2024 02:42:13 GMT+0800 (China Standard Time)

I've been trying to test that using strace ..

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ strace -fe trace=file bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o
execve("/usr/sbin/bpftool", ["bpftool", "--debug", "gen", "object", "procdump.ebpf.o", "procdump_ebpf.o"], 0x7fff868ff1f8 /* 30 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "procdump.ebpf.o", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
libbpf: linker: adding object file 'procdump_ebpf.o'...
openat(AT_FDCWD, "procdump_ebpf.o", O_RDONLY|O_CLOEXEC) = 4
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)
+++ exited with 234 +++

So noting is trying to access to sysfs so kernel version and availability of the /sys/kernel/btf/vmlinux should not be relevant.

Tomasz Kłoczko · Answer 6 · Sat Apr 20 2024 02:45:52 GMT+0800 (China Standard Time)

I've tested as well use bpdtools instead from package generated by me use package fro Fedora rawhide and result is the same

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ rpm -qf /usr/sbin/bpftool; echo
bpftool-6.9.0-0.rc4.20240416git96fca68c4fbf7.38.fc41.x86_64

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o
libbpf: linker: adding object file 'procdump_ebpf.o'...
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)

Tomasz Kłoczko · Answer 7 · Sat Apr 20 2024 02:53:17 GMT+0800 (China Standard Time)

Here is full strace output

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ strace -f bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o
execve("/usr/sbin/bpftool", ["bpftool", "--debug", "gen", "object", "procdump.ebpf.o", "procdump_ebpf.o"], 0x7fffdb955750 /* 30 vars */) = 0
brk(NULL)                               = 0x560ae84cb000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3dab9000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=30649, ...}) = 0
mmap(NULL, 30649, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3c3dab1000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=111824, ...}) = 0
mmap(NULL, 110608, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da95000
mmap(0x7f3c3da98000, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f3c3da98000
mmap(0x7f3c3daab000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f3c3daab000
mmap(0x7f3c3daaf000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f3c3daaf000
mmap(0x7f3c3dab0000, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3dab0000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=141048, ...}) = 0
mmap(NULL, 139416, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da72000
mmap(0x7f3c3da74000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f3c3da74000
mmap(0x7f3c3da8b000, 32768, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f3c3da8b000
mmap(0x7f3c3da93000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f3c3da93000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0,q\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=44920, ...}) = 0
mmap(NULL, 45136, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da66000
mmap(0x7f3c3da69000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f3c3da69000
mmap(0x7f3c3da6e000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f3c3da6e000
mmap(0x7f3c3da70000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7f3c3da70000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\243\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
fstat(3, {st_mode=S_IFREG|0755, st_size=2433096, ...}) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2022296, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3d878000
mmap(0x7f3c3d8a0000, 1482752, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f3c3d8a0000
mmap(0x7f3c3da0a000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x192000) = 0x7f3c3da0a000
mmap(0x7f3c3da58000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e0000) = 0x7f3c3da58000
mmap(0x7f3c3da5e000, 31640, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3da5e000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=786912, ...}) = 0
mmap(NULL, 782384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3d7b8000
mmap(0x7f3c3d7bc000, 708608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f3c3d7bc000
mmap(0x7f3c3d869000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb1000) = 0x7f3c3d869000
mmap(0x7f3c3d876000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbd000) = 0x7f3c3d876000
mmap(0x7f3c3d877000, 48, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d877000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d7b6000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d7b4000
arch_prctl(ARCH_SET_FS, 0x7f3c3d7b7640) = 0
set_tid_address(0x7f3c3d7b7910)         = 1146977
set_robust_list(0x7f3c3d7b7920, 24)     = 0
rseq(0x7f3c3d7b7f60, 0x20, 0, 0x53053053) = 0
mprotect(0x7f3c3da58000, 16384, PROT_READ) = 0
mprotect(0x7f3c3d876000, 4096, PROT_READ) = 0
mprotect(0x7f3c3da70000, 4096, PROT_READ) = 0
mprotect(0x7f3c3da93000, 4096, PROT_READ) = 0
mprotect(0x7f3c3daaf000, 4096, PROT_READ) = 0
mprotect(0x560ae6e3b000, 765952, PROT_READ) = 0
mprotect(0x7f3c3daed000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7f3c3dab1000, 30649)           = 0
prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 0
prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
getrandom("\xb7\xbc\x71\xd7\x8a\xa7\x44\x63", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x560ae84cb000
brk(0x560ae84ec000)                     = 0x560ae84ec000
openat(AT_FDCWD, "procdump.ebpf.o", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
write(2, "libbpf: linker: adding object fi"..., 56libbpf: linker: adding object file 'procdump_ebpf.o'...
) = 56
openat(AT_FDCWD, "procdump_ebpf.o", O_RDONLY|O_CLOEXEC) = 4
fcntl(4, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
fstat(4, {st_mode=S_IFREG|0644, st_size=27104, ...}) = 0
pread64(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0\367\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 64, 0) = 64
pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2432, 24672) = 2432
pread64(4, "\0uprobe_reallocarray\0uretprobe_r"..., 2579, 22088) = 2579
pread64(4, "{\32\320\377\0\0\0\0y\241\320\377\0\0\0\0{\32\360\377\0\0\0\0\30\1\0\0\0\0\0\0"..., 2712, 64) = 2712
pread64(4, "\30\0\0\0\0\0\0\0\1\0\0\0\332\0\0\0`\0\0\0\0\0\0\0\1\0\0\0\333\0\0\0"..., 144, 20680) = 144
pread64(4, "{\32\250\377\0\0\0\0y\241\250\377\0\0\0\0y\22h\0\0\0\0\0{*\330\377\0\0\0\0"..., 1024, 2776) = 1024
pread64(4, "\200\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\270\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 20824) = 128
pread64(4, "{\32\310\377\0\0\0\0y\241\310\377\0\0\0\0y\21P\0\0\0\0\0{\32\360\377\0\0\0\0"..., 608, 3800) = 608
pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 20952) = 64
pread64(4, "{\32\270\377\0\0\0\0y\241\270\377\0\0\0\0y\21p\0\0\0\0\0{\32\340\377\0\0\0\0"..., 728, 4408) = 728
pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\343\0\0\0"..., 64, 21016) = 64
pread64(4, "{\32\310\377\0\0\0\0y\241\310\377\0\0\0\0y\21p\0\0\0\0\0{\32\360\377\0\0\0\0"..., 608, 5136) = 608
pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21080) = 64
pread64(4, "{\32\230\377\0\0\0\0y\242\230\377\0\0\0\0y!p\0\0\0\0\0{*\250\377\0\0\0\0"..., 1056, 5744) = 1056
pread64(4, "\240\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21144) = 128
pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 6800) = 632
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21272) = 64
pread64(4, "{\32\250\377\0\0\0\0y\242\250\377\0\0\0\0y!P\0\0\0\0\0{*\270\377\0\0\0\0"..., 752, 7432) = 752
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\343\0\0\0"..., 64, 21336) = 64
pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 8184) = 632
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21400) = 64
pread64(4, "{\32\220\377\0\0\0\0y\243\220\377\0\0\0\0y2p\0\0\0\0\0y1h\0\0\0\0\0"..., 1072, 8816) = 1072
pread64(4, "\260\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\350\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21464) = 128
pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 9888) = 632
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21592) = 64
pread64(4, "{\32\220\377\0\0\0\0y\243\220\377\0\0\0\0y2p\0\0\0\0\0y1h\0\0\0\0\0"..., 1072, 10520) = 1072
pread64(4, "\260\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\350\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21656) = 128
pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 11592) = 632
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21784) = 64
pread64(4, "{\32\210\377\0\0\0\0y\244\210\377\0\0\0\0yCp\0\0\0\0\0yBh\0\0\0\0\0"..., 1088, 12224) = 1088
pread64(4, "\300\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\370\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21848) = 128
pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 13312) = 632
pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21976) = 64
pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 144, 13944) = 144
pread64(4, "Dual BSD/GPL\0", 13, 14088) = 13
pread64(4, "[ResourceAllocHelper] Failed: Ge"..., 718, 14104) = 718
pread64(4, "\337\1\345\1\347\1\350\1\351\1\352\1\353\1\354\1\355\1\356\1\357\1\360\1\361\1\362\1\334\1\363\1"..., 48, 22040) = 48
pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0K\3\0\0\4\0\361\377"..., 5856, 14824) = 5856
brk(0x560ae850e000)                     = 0x560ae850e000
brk(0x560ae850d000)                     = 0x560ae850d000
write(2, "libbpf: failed to find BTF info "..., 61libbpf: failed to find BTF info for object 'procdump_ebpf.o'
) = 61
close(4)                                = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "failed to link 'procdump_ebpf.o'"..., 55failed to link 'procdump_ebpf.o': Invalid argument (22)) = 55
write(2, "\n", 1
)                       = 1
close(3)                                = 0
brk(0x560ae8509000)                     = 0x560ae8509000
exit_group(-22)                         = ?
+++ exited with 234 +++

after delete procdump_ebpf.o':

```console [ 27%] Building EBPF object procdump_ebpf.o clang -nostdinc -isystem `gcc -print-file-name=include` -I /usr/include -I /usr/include/x86_64-linux-gnu -I /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 -I /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu -I /usr/include -D __KERNEL__ -D __BPF_TRACING__ -D __TARGET_ARCH_x86 -D __linux__ -target bpf -fno-stack-protector -c /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c -o procdump_ebpf.o /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:20: warning: left operand of comma operator has no effect [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:103: warning: left operand of comma operator has no effect [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:110: warning: expression result unused [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:20: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:120: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:134: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:141: warning: expression result unused [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:20: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:120: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:134: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:141: warning: expression result unused [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:16: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:103: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:117: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:124: warning: expression result unused [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:20: warning: left operand of comma operator has no effect [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:104: warning: left operand of comma operator has no effect [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:111: warning: expression result unused [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:20: warning: left operand of comma operator has no effect [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:105: warning: left operand of comma operator has no effect [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:112: warning: expression result unused [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:16: warning: left operand of comma operator has no effect [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:87: warning: left operand of comma operator has no effect [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:94: warning: expression result unused [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:20: warning: left operand of comma operator has no effect [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:110: warning: left operand of comma operator has no effect [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:116: warning: expression result unused [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:20: warning: left operand of comma operator has no effect [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:111: warning: left operand of comma operator has no effect [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:117: warning: expression result unused [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:16: warning: left operand of comma operator has no effect [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:93: warning: left operand of comma operator has no effect [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:99: warning: expression result unused [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ 33 warnings generated. [ 30%] Generating procdump.ebpf.o /usr/sbin/bpftool gen object procdump.ebpf.o procdump_ebpf.o libbpf: failed to find BTF info for object 'procdump_ebpf.o' Error: failed to link 'procdump_ebpf.o': Invalid argument (22) make[2]: *** [CMakeFiles/procdump_ebpf.dir/build.make:78: procdump.ebpf.o] Error 234 make[2]: *** Deleting file 'procdump.ebpf.o' make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu' make[1]: *** [CMakeFiles/Makefile2:308: CMakeFiles/procdump_ebpf.dir/all] Error 2 make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu' make: *** [Makefile:94: all] Error 2 ```

So clang (18.1.4) generates that object file and than /usr/sbin/bpftool cannot link it.
What about use gcc instead clang? In this case I'm playing with gcc from Fedora rawhide which IIRC should be able to build bpf code as well 🤔
(I don't know however how to do that)

Mario Hewardt · Answer 8 · Mon Apr 22 2024 23:06:28 GMT+0800 (China Standard Time)

Thanks. I think GCC can build BPF code nowadays but it's not something I have tried or is supported at the moment. Feel free to try it out if you want. There are a couple of other things to check:

Does your kernel config have BPF/BTF enabled? It should but might be good to check.
Do BPF programs run on your system in general? You can try building and running a simple one from here - https://github.com/libbpf/libbpf-bootstrap

Tomasz Kłoczko · Answer 9 · Mon Apr 22 2024 23:21:30 GMT+0800 (China Standard Time)

As I wrote I'm using gcc from Fedora and looking on spec file I suppose as well that exactly those binaries should be able to compile bpf code and cannot only find details how to do that.
Yes kernel support BFP/BTF.
Will try later with libbpf-bootstrap 🤔

Mario Hewardt · Answer 10 · Tue Apr 23 2024 00:29:47 GMT+0800 (China Standard Time)

Not sure if this might help - https://gcc.gnu.org/wiki/BPFBackEnd

Tomasz Kłoczko · Answer 11 · Tue Apr 23 2024 01:06:50 GMT+0800 (China Standard Time)

I've saw already that page and it cannot be used in context of Fedora binaries because fedora uses multiarch gcc/binutils setup and is not using crosscompilers for bpf.
If I'll not find exact solution I'll try soon to open fedora ticket with question because I have very similar issue with dtrace oracle/dtrace-utils#71

Elena Zannoni · Answer 12 · Tue Apr 23 2024 04:16:24 GMT+0800 (China Standard Time)

If Fedora would package the crosses for BPF that would be really nice and solve all these issues indeed.

Mario Hewardt · Answer 13 · Tue Apr 23 2024 04:42:38 GMT+0800 (China Standard Time)

In order to support memory leak tracking (using -restrack) switch, ProcDump relies on eBPF. One option is to remove the libbpf dependencies from the build (behind a new def, NO_RESTRACK or some such). This would avoid build failures for distros without proper eBPF support. Of course, this also means that the -restrack switch would not work on those builds.

Tomasz Kłoczko · Answer 14 · Tue Apr 23 2024 05:31:31 GMT+0800 (China Standard Time)

In order to support memory leak tracking (using -restrack) switch, ProcDump relies on eBPF

That technique is known more than decade (since Solaris DTrace is known which Linux eBPF only mimics).

Mario Hewardt · Answer 15 · Tue Apr 23 2024 06:24:48 GMT+0800 (China Standard Time)

I'm not making any statements as to which technology came first or if one is better than the other :) Suffice to say, I was presenting an optional build strategy if you don't need memory leak tracking and hence can remove the dependency on eBPF.