Compile issue: "invalid relo against 'nr_latches'"
GoWest279 opened this issue · comments
Hi, colleagues
I'm trying to compile the ghost-userspace application, but come across the
libbpf: prog 'sched_ghost_latched': invalid relo against 'nr_latches' in special section 0xfff2; forgot to initialize global var?..
Error: failed to open BPF object file: Relocation failed
I compiled and installed the ghost kernel and commented the redefined variables in the agent.h according to the posts in #3.
My setup is:
$uname -a
Linux ubuntu 5.11.0+ #2 SMP Sat Jan 15 18:10:04 PST 2022 x86_64 x86_64 x86_64 GNU/Linux
$gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Any idea will be appreciated!
Full debug message is as follows:
ghost@ubuntu:~/ghost-userspace$ bazel build -c opt ... --sandbox_debug
DEBUG: Rule 'linux' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "2993dcd082a5f14d912d7dd60eec7266c3e2560b", shallow_since = "1642200822 -0800" and dropping ["branch"]
DEBUG: Repository linux instantiated at:
/home/ghost/ghost-userspace/WORKSPACE:9:19: in <toplevel>
Repository rule new_git_repository defined at:
/home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/external/bazel_tools/tools/build_defs/repo/git.bzl:186:37: in <toplevel>
INFO: Analyzed 62 targets (0 packages loaded, 0 targets configured).
INFO: Found 62 targets...
ERROR: /home/ghost/ghost-userspace/BUILD:455:13: Executing genrule //:schedghostidle_bpf_skel failed: (Exit 255): linux-sandbox failed: error executing command
(cd /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost && \
exec env - \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
TMPDIR=/tmp \
/home/ghost/.cache/bazel/_bazel_ghost/install/64841bf12de13c7518c7ada0994bafe2/linux-sandbox -t 15 -w /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost -w /tmp -w /dev/shm -D -- /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-opt/bin/external/linux/bpftool/bin/bpftool gen skeleton bazel-out/k8-opt/bin/third_party/bpf/schedghostidle_bpf.o > bazel-out/k8-opt/bin/bpf/user/schedghostidle_bpf.skel.h && sed -i '\''s/#include <bpf/#include <libbpf/g'\'' bazel-out/k8-opt/bin/bpf/user/schedghostidle_bpf.skel.h')
1642306174.978625595: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1642306174.978645467: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1642306174.978909323: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 43757
1642306174.978923332: src/main/tools/linux-sandbox-pid1.cc:447: Pid1Main started
1642306174.978995530: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1642306174.979195929: src/main/tools/linux-sandbox-pid1.cc:189: working dir: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979232495: src/main/tools/linux-sandbox-pid1.cc:221: writable: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979239022: src/main/tools/linux-sandbox-pid1.cc:221: writable: /tmp
1642306174.979252980: src/main/tools/linux-sandbox-pid1.cc:221: writable: /dev/shm
1642306174.979306772: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /
1642306174.979319554: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev
1642306174.979325888: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/pts
1642306174.979331796: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1642306174.979336313: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/hugepages
1642306174.979340338: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /dev/mqueue
1642306174.979344049: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run
1642306174.979347580: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/lock
1642306174.979351286: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/vmblock-fuse
1642306174.979354975: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000
1642306174.979359065: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000/gvfs
1642306174.979363937: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /run/user/1000/doc
1642306174.979394596: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys
1642306174.979402324: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/security
1642306174.979411365: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup
1642306174.979417473: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/unified
1642306174.979424332: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/systemd
1642306174.979430311: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/rdma
1642306174.979436364: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/freezer
1642306174.979443342: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/net_cls,net_prio
1642306174.979449538: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/hugetlb
1642306174.979455775: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpuset
1642306174.979461877: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/perf_event
1642306174.979468629: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/memory
1642306174.979474932: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/blkio
1642306174.979504823: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/cpu,cpuacct
1642306174.979513175: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/pids
1642306174.979520327: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/cgroup/devices
1642306174.979527392: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/pstore
1642306174.979533629: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/bpf
1642306174.979539387: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/debug
1642306174.979546065: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/tracing
1642306174.979552647: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/fs/fuse/connections
1642306174.979559791: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /sys/kernel/config
1642306174.979574151: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc
1642306174.979580263: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc/sys/fs/binfmt_misc
1642306174.979586460: src/main/tools/linux-sandbox-pid1.cc:311: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1642306174.979599086: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /proc/sys/fs/binfmt_misc
1642306174.979605623: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/bare/5
1642306174.979609466: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/1705
1642306174.979624779: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core18/2284
1642306174.979627969: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/core20/1270
1642306174.979630692: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-34-1804/24
1642306174.979634116: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-38-2004/87
1642306174.979638038: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gnome-3-34-1804/77
1642306174.979695993: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gtk-common-themes/1519
1642306174.979704754: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/gtk-common-themes/1506
1642306174.979709121: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snap-store/558
1642306174.979713496: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snap-store/433
1642306174.979717902: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /boot/efi
1642306174.979725429: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/14295
1642306174.979729834: src/main/tools/linux-sandbox-pid1.cc:291: remount ro: /snap/snapd/7264
1642306174.979734562: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979744910: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /home/ghost/.cache/bazel/_bazel_ghost/24bbcc1ee3641e3e472020f868310314/sandbox/linux-sandbox/1199/execroot/com_google_ghost
1642306174.979750062: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /tmp
1642306174.979754576: src/main/tools/linux-sandbox-pid1.cc:291: remount rw: /dev/shm
1642306174.979801257: src/main/tools/linux-sandbox-pid1.cc:376: calling fork...
1642306174.979938087: src/main/tools/linux-sandbox-pid1.cc:406: child started with PID 2
libbpf: prog 'sched_ghost_latched': invalid relo against 'nr_latches' in special section 0xfff2; forgot to initialize global var?..
Error: failed to open BPF object file: Relocation failed
1642306174.983345378: src/main/tools/linux-sandbox-pid1.cc:423: wait returned pid=2, status=0xff00
1642306174.983354385: src/main/tools/linux-sandbox-pid1.cc:441: child exited normally with code 255
1642306174.984785169: src/main/tools/linux-sandbox.cc:233: child exited normally with code 255
INFO: Elapsed time: 0.408s, Critical Path: 0.01s
INFO: 8 processes: 8 internal.
FAILED: Build did NOT complete successfully
Hello,
me I have meet this error and to solve this problem, I have just initialize some variable in
the following file : third_party/bpf/schedghostidle.bpf.c.
In that file I have just initialize the following variable:
uint64_t nr_latches = 0;
uint64_t nr_bpf_latches = 0;
uint64_t nr_idle_to_bpf_latches = 0;
Try to do it and we will see if it work on your project.
@NGUETOUM Thanks so much!
It solved the compile problem. Just not sure whether those variables should be initialized to specific values. I am going to try some example benchmark to see what will happen:)
@john-liu2039 They should be initialized to 0. I need to look into why this is causing compile issues on your end since the global variables should be zero-initialized by default just as they would be in a userspace program. What version of clang are you using?
Thanks, @jackhumphries :) My clang setup is as follows:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
BTW, I set up the ghost kernel and userspace with the VMware virtual machine.
Thanks so much for your help!
Thanks, I'll take a look tomorrow. If it turns out the zero-initialization isn't guaranteed by the standard, then I'll push a commit to fix this. Otherwise I'll figure out what the real issue is and post here.
probably due to a change in libbpf. perhaps you're using a newer one than we are. i see other some commits in libbpf-tools zero-initializing globals.
the fix would be to zero initialize the globals in all of our tools.
Thanks Barret.
I have explicitly initialized the BPF global variables to 0 in our internal Google codebase. This will make its way out to the open source codebase soon.