Giters

google / ghost-userspace

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Issues compiling

kuroro99 opened this issue · comments

commented

Hi,
I've compiled the kernel and replaced my previous kernel in my ubuntu, then when using the command bazel build -c opt ... , I got this error

                 from lib/enclave.cc:24:
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:244:6: error: use of enum 'bpf_stats_type' without previous declaration
  244 | enum bpf_stats_type; /* defined in up-to-date linux/bpf.h */
      |      ^~~~~~~~~~~~~~
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38: error: use of enum 'bpf_stats_type' without previous declaration
  245 | LIBBPF_API int bpf_enable_stats(enum bpf_stats_type type);
      |                                      ^~~~~~~~~~~~~~
Target //:agent_test failed to build

This may mean we didn't declared the type of enum of C++, but in
bazel-out/k8-fastbuild/bin/external/linux/libbpf/include/libbpf/bpf.h:245:38
there declared the using of extern C

extern "C" {
#endif

And here are the environments:
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04)
bazel 4.2.1
Could you give my some suggestions?
Thank you

commented

Hi,

enum bpf_stats_type is declared in the kernel tree in include/uapi/linux/bpf.h. I believe what you're referring to in libbpf/bpf.h is just a forward declaration.

A few questions:

  1. Can you verify that you booted the ghOSt kernel (what is the output of uname -a)?
  2. Can you see the declaration for enum bpf_stats_type in /usr/src/linux-headers-VERSION/include/uapi/linux/bpf.h? Replace VERSION in the path with the ghOSt kernel version (5.11 or so).

Let me know and then I can provide further assistance. Thanks.

commented

Thank you for the response! And when I didn't compile the kernel , my grub menu just have the version 5.11.0-27-generic , then when have compiled the kernel , my kernel version is 5.11.0-40-generic . So I believe the ghOSt kernel is booted. (I didn't use the suggested version 5.40 as the kernel when ghOSt not compiled, does it necessary?
About the declaration, I found the declaration of enum bpf_stats_type in /usr/src/linux-headers-5.11.0-40-generic/include/uapi/linux/bpf.h :

enum bpf_stats_type {
        /* enabled run_time_ns and run_cnt */
        BPF_STATS_RUN_TIME = 0,
};

So, I wonder why can't find the previous declaration of the enum bpf_stats_type in the libbpf/bpf.h , it just includes the linux/bpf.h .

commented

Well , I found that it's because I did not install the header file of ghOSt to the directory usr/include/ , ,,now it's solved, thanks for your respone again,,,I am sorry for this stupid question

commented

Hi @jackhumphries, I actually didn't understand how @mrkrl solved the problem! Should we run make headers_install? I did but no success.

For my case, I downloaded the ghost_kernel and then cp boot/config <mydir>/.config && make oldconfig && make headers_install && make modules_install && make install && reboot. However, no /usr/src/linux-headers-5.11.0+ had been created in my directories.

I think all these headers should go to /usr/include but make headers_install seems not doing it!

Here is my uname -a: Linux *.cloudlab.us 5.11.0+ #3 SMP Sun Dec 12 19:37:12 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

Thanks.

commented

Hi @sarsanaee , I think you should move the files the command make headers_install generated in ghost-userspace/usr/include to your reference path /usr/include then it may work well.

commented

Hi all

Yeah, thank you @mrkrl I think that helps but doesn't solve the problem totally.

I believe the problem mostly boils down to include/linux/bpf.h. There are some ghost variables that are defined in that file and then get redefined in the agent.h or agent.c again. I think there is a sort of miss-match between the latest userspace and the kernel available unless I'm missing something.

However, these all are immediately solvable in my understanding. You either comment duplicated ones from agent.c/agent.h or from bpf.h. But I'm suspecting that those things should stay in bpf.h files as developers might need them in the ebpf programs.

Thanks,
Alireza

commented

@sarsanaee Delete the variables redefined in agent.h may solve the compile problem. And you may have some other problems in runing which need to change some variables in agent.h ,you can see in #2

commented

@mrkrl So you suggest leaving these vars in agent.h instead of bpf.h?

commented

@sarsanaee I am sorry that I said something ambiguity,,,,So,I mean deleting the vars in agent.h and leaving them in bpf.h.I solved this problem by this way because I think it's better to change the ghost-userspace's code rather than the kernel's.

commented

@mrkrl This makes sense.

My only concern here is that how you're gonna expose ghost's vars to the bpf programs then? Maybe there is another way to do so! If things are available in bpf.h we can still play with them at bpf hooks. My hunch is that the main reason we are running into this issue is the fact that they initally wanted to vars on both bpf and user sides which is sensical, however, we run into some multiple include kind of problem now.

But again, maybe I'm missing something.

commented

Hi both,

Thanks to @sarsanaee, I was able to log into a machine with this issue. Here is what you need to do to resolve this issue, from start to finish:

  1. Make sure you install the kernel headers. From the kernel tree, run make headers_install INSTALL_HDR_PATH=/usr. This will resolve the use of enum 'bpf_stats_type' without previous declaration issue since the header with that definition is now visible to the userspace code. In the future, I will update the kernel repo with compile and install instructions to make this clear.

  2. The second issue about ghOSt variables getting redefined is a bug in our open source project. We use a different compile environment inside of Google, so we have to convert the code before pushing to the open source repo and it looks like some things didn't get stripped out. I will fix this in a forthcoming commit, but for now you should do the following:

In bpf/user/agent.h, delete the following code:

// From include/uapi/linux/bpf.h for the ghost kernel.

struct bpf_ghost_sched {};

enum {
  BPF_PROG_TYPE_GHOST_SCHED = 35,

  BPF_GHOST_SCHED_SKIP_TICK = 50,
  BPF_GHOST_SCHED_PNT,
  BPF_GHOST_MAX_ATTACH_TYPE,  // __MAX_BPF_ATTACH_TYPE
};

// end include/uapi/linux/bpf.h

And in place of the code you deleted, write the following:

#define BPF_GHOST_MAX_ATTACH_TYPE __MAX_BPF_ATTACH_TYPE

Everything should compile now. Let me know if any other issues come up and thanks for pointing this out!

commented

@jackhumphries
Hi, Jack,
I'm trying to follow the compile procedure and also came across the "redfine" issue.
I followed your patch above, but it came out new issue:

ERROR: /home/ghost/ghost-userspace/BUILD:33:11: Compiling bpf/user/agent.c failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 27 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
bpf/user/agent.c:47:17: error: 'BPF_GHOST_MAX_ATTACH_TYPE' undeclared here (not in a function); did you mean 'BPF_GHOST_SCHED_MAX_ATTACH_TYPE'?
   47 | } bpf_registry [BPF_GHOST_MAX_ATTACH_TYPE];
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~
      |                 BPF_GHOST_SCHED_MAX_ATTACH_TYPE
bpf/user/agent.c:47:3: warning: 'bpf_registry' defined but not used [-Wunused-variable]
   47 | } bpf_registry [BPF_GHOST_MAX_ATTACH_TYPE];
      |   ^~~~~~~~~~~~
INFO: Elapsed time: 0.455s, Critical Path: 0.05s
INFO: 9 processes: 9 internal.
FAILED: Build did NOT complete successfully

Did I miss anything here?
Should the patch also modify the agent.c?

Thanks so much!

commented

Oh sorry -- I made a typo in that comment. Try with the updated code in that comment and it should work.

commented

Hi @jackhumphries

It seems I should upgrade my gcc and g++ now!

alireza@test:~/ghost-userspace$ bazel build -c opt ...
INFO: Analyzed 75 targets (68 packages loaded, 78163 targets configured).
INFO: Found 75 targets...
ERROR: /home/alireza/.cache/bazel/_bazel_alireza/789238d1a6b81fb3921961afbd0ce4d4/external/com_google_absl/absl/random/internal/BUILD.bazel:304:11: Compiling absl/random/internal/randen_detect.cc failed: (Exit 1): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 36 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
gcc: error: unrecognized command line option '-std=c++20'; did you mean '-std=c++2a'?
INFO: Elapsed time: 9.377s, Critical Path: 0.51s
INFO: 145 processes: 145 internal.
FAILED: Build did NOT complete successfully

I also changed the .bazelrc to change the standard to c++2a which is something understandable to GCC9 apparently but I ended up not finding #include <latch> in simple_exp.cc test.

But when I installed gcc-11 and g++-11, I sort of managed to compile the ghost-userspace at the end of the day :)

Further, I got this warning as well:

INFO: From Executing genrule //:biff_bpf_skel:
libbpf: elf: skipping unrecognized data section(6) .rodata.str1.1