wrappers/ld fails on `-plugin`

Question

wrappers/ld fails on `-plugin`

siedentop opened this issue 5 years ago · comments

Christoph Siedentop commented 5 years ago

It looks like the -plugin switch is not understood by wrappers/ld and hence crashes gg.

I was trying to build LLVM with gg.

From opening the core dump with gdb:
Core was generated by model-ld -plugin /usr/lib/gcc/x86_64-linux-gnu/7/liblto_plugin.so -plugin-opt=/'.
`

Question 1: How do I enable debug symbols for gg, such that the core dump is actually helpful?

Here is what man ld has to say about -plugin:

 -plugin name
      Involve a plugin in the linking process.  The name parameter is the absolute filename of the plugin.  Usually this parameter is automatically added by the complier, when using link time optimization,
      but users can also add their own plugins if they so wish.

      Note that the location of the compiler originated plugins is different from the place where the ar, nm and ranlib programs search for their plugins.  In order for those commands to make use of a
     compiler based plugin it must first be copied into the ${libdir}/bfd-plugins directory.  All gcc based linker plugins are backward compatible, so it is sufficient to just copy in the newest one.

Note, LTO was disabled in the CMake config for LLVM by default and has stayed like that.

ubuntu@ip-172-31-14-212:~/build/llvm$ gg infer ninja -j$(nproc)
[183/3942] Generating VCSRevision.h
-- Found Git: /usr/bin/git (found version "2.17.1")
[213/3942] Linking CXX executable bin/llvm-tblgen
FAILED: bin/llvm-tblgen
: && /usr/bin/c++  -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3  -Wl,-allow-shlib-undefined    -Wl,-rpath-link,/home/ubuntu/build/llvm/./lib  -Wl,-O3 -Wl,--gc-sections utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterInst.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Attributes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CallingConvEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeEmitterGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenHwModes.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenInstruction.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenMapTable.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenRegisters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenSchedule.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenTarget.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcherOpt.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DAGISelMatcher.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DFAPacketizerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/DisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/ExegesisEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FastISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/GlobalISelEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InfoByHwMode.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrDocsEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/IntrinsicEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/OptParserEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PredicateExpander.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/PseudoLoweringEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RISCVCompressInstEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterBankEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterInfoEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SDNodeProperties.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SearchableTableEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/SubtargetFeatureInfo.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/TableGen.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/Types.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86DisassemblerTables.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86EVEX2VEXTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86FoldTablesEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86ModRMFilters.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/X86RecognizableInstr.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/WebAssemblyDisassemblerEmitter.cpp.o utils/TableGen/CMakeFiles/llvm-tblgen.dir/CTagsEmitter.cpp.o  -o bin/llvm-tblgen  -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.a lib/libLLVMTableGen.a -lpthread lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm lib/libLLVMDemangle.a && :
terminate called after throwing an instance of 'std::runtime_error'
  what():  unknown option: -plugin
/home/ubuntu/gg/src/models/wrappers/ld: line 2:   306 Aborted                 (core dumped) model-ld "$@"
collect2: error: ld returned 134 exit status
[220/3942] Building CXX object lib/MC/CMakeFiles/LLVMMC.dir/ELFObjectWriter.cpp.o
ninja: build stopped: subcommand failed.

My Bash history (from a fresh Ubuntu 18.04 EC2 instance):

Removed non-relevant lines

   5  sudo apt update
    6  sudo apt-get install gcc-7 g++-7 protobuf-compiler libprotobuf-dev                      libcrypto++-dev libcap-dev                      libncurses5-dev libboost-dev libssl-dev autopoint help2man                      libhiredis-dev texinfo automake libtool pkg-config
    7  ./fetch-submodules.sh
    8  ./autogen.sh
    9  ./configure
   10  make -j$(nproc)
   11  sudo make install
   12  vim environment.sh
   20  source environment.sh
   21  cd src/remote/
   24  sudo apt install python3-boto3
   26  make ggfunctions
   # Snip trying out, successfully, the mosh demo.
   28  git clone https://github.com/mobile-shell/mosh
   # ...
   48  gg force --jobs 100 --engine lambda src/frontend/mosh-server
   
   # Now to LLVM!
   51  git clone https://github.com/llvm/llvm-project.git
   52  mkdir build
   53  cd build/llvm
   62  cmake -GNinja ~/llvm-project/llvm -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=On
   63  gg init
   65  gg infer ninja -j$(nproc)

Sadjad Fouladi commented 5 years ago

#23

Christoph Siedentop · Answer 1 · Fri Aug 09 2019 23:09:25 GMT+0800 (China Standard Time)

Notes & Updates:

After patching src/models/ld.cc it will also complain about -plugin-opt, --sysroot, --eh-frame-hdr. Patching all these lead to stat() errors.
Calling already cmake as gg infer cmake ... seems to help.
Not using Ninja (but instead make) seems to help, too. (Probably more so)

This still leaves this:

[ 16%] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/OptParserEmitter.cpp.o
terminate called after throwing an instance of 'unix_error'
  what():  stat: No such file or directory
/home/ubuntu/gg/src/models/wrappers/ld: line 2: 32432 Aborted                 (core dumped) model-ld "$@"
collect2: error: ld returned 134 exit status
utils/PerfectShuffle/CMakeFiles/llvm-PerfectShuffle.dir/build.make:94: recipe for target 'bin/llvm-PerfectShuffle' failed
make[2]: *** [bin/llvm-PerfectShuffle] Error 1
CMakeFiles/Makefile2:19614: recipe for target 'utils/PerfectShuffle/CMakeFiles/llvm-PerfectShuffle.dir/all' failed
make[1]: *** [utils/PerfectShuffle/CMakeFiles/llvm-PerfectShuffle.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Nick S. · Answer 2 · Mon Aug 12 2019 23:12:18 GMT+0800 (China Standard Time)

@siedentop have you try to build llvm without any flags?

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build_gg && cd build_gg
gg init
gg infer cmake ./../llvm
gg infer cmake --build .

Btw, I'm not sure that gg will be ok with llvm due to cmake. But currently, the build process is reaching > 23% completion and still working (it is running on my laptop, so it's not so fast).

Christoph Siedentop · Answer 3 · Tue Aug 13 2019 09:32:11 GMT+0800 (China Standard Time)

Thanks, @ssnickolay for taking a look.

I had tried gg init; gg infer cmake ... ; gg infer make ... before, but now tried your variant. I only get to 8%.

It compiles fine past the 8% building locally, so it's not a dependency which is missing.

  188  git clone https://github.com/llvm/llvm-project
  189  cd llvm-project/
  190  mkdir build_gg
  191  cd build_gg
  192  gg init
  193  gg infer cmake ./../llvm
  194  source ~/src/gg/environment.sh
  195  gg infer cmake ./../llvm
  196  gg infer cmake --build .

The build step fails with the following.

...
Scanning dependencies of target llvm-tblgen
[  8%] Linking CXX executable ../../bin/llvm-tblgen
╭╼ generating model for g++
├─ linked: TYgQkWjMxZQiDlekeM6Vk_FyaNImIejV4MbvTshfSM4Y000028b5
╰╼ output: ../../bin/llvm-tblgen
[  8%] Built target llvm-tblgen
Scanning dependencies of target AttributeCompatFuncTableGen
[  8%] Building AttributesCompatFunc.inc...
→ Loading the thunks...  done (52 ms).
In file included from /home/ubuntu/llvm-project/llvm/lib/Support/Process.cpp:93:0:
/home/ubuntu/llvm-project/llvm/lib/Support/Unix/Process.inc: In static member function 'static unsigned int llvm::sys::Process::GetRandomNumber()':
/home/ubuntu/llvm-project/llvm/lib/Support/Unix/Process.inc:454:10: error: 'arc4random' was not declared in this scope
/home/ubuntu/llvm-project/llvm/lib/Support/Unix/Process.inc:454:10: note: suggested alternative: 'srandom'
rmdir /tmp/thunk-execute.V3u3PY: Directory not empty
std::exception
 `T242qEgXj1ATQwnyXqbhLjZYFs7gWbVFVuZNvJVaoX5E000003dd': process exited with failure status 1
gg-force: `T242qEgXj1ATQwnyXqbhLjZYFs7gWbVFVuZNvJVaoX5E000003dd': process exited with failure status 5
gg-force-and-run: `gg-force': process exited with failure status 1
lib/IR/CMakeFiles/AttributeCompatFuncTableGen.dir/build.make:93: recipe for target 'lib/IR/AttributesCompatFunc.inc' failed
make[2]: *** [lib/IR/AttributesCompatFunc.inc] Error 1
CMakeFiles/Makefile2:1710: recipe for target 'lib/IR/CMakeFiles/AttributeCompatFuncTableGen.dir/all' failed
make[1]: *** [lib/IR/CMakeFiles/AttributeCompatFuncTableGen.dir/all] Error 2
Makefile:151: recipe for target 'all' failed
make: *** [all] Error 2

Sadjad Fouladi · Answer 4 · Tue Aug 13 2019 10:07:45 GMT+0800 (China Standard Time)

Hi @siedentop and @ssnickolay,

Thank you for opening this issue, and of course for trying out gg!

One important concept in using gg for compilation is the separation between generating the dependency graph for the build (the part done with gg infer) and doing the actual work (the part done with gg force). LLVM build system is a bit complicated for gg for two reasons:

LLVM uses CMake, which hardcodes the path to gcc in the Makefiles and that makes it a bit troublesome for gg to do the inference (which relies on manipulating the PATH). One workaround is to run cmake as follows:

gg init
CXX="g++" CC="gcc" GG_BYPASS=1 gg infer cmake <SRC-PATH>

This will generate the Makefiles that can be used to do the inference later.

As a part of the build process, LLVM build system compiles a binary, and then proceeds to execute that binary to generate some header files to continue the build. This kinda breaks that separation concept I mentioned earlier. The workaround for this is to first build that binary, and then compile LLVM.

To compile the binary, after step (1), run the following commands:

gg infer make -j`nproc` llvm-tblgen
gg force --jobs 1200 --engine lambda bin/llvm-tblgen

And now, you can infer the whole thing:

gg infer make -j`nproc`

If you look at the build/bin/ folder, you can see the target binaries. To compile, for example llvm-ar, run:

gg force --jobs 2000 --engine lambda bin/llvm-ar

Thanks again for giving gg a try. We understand that this is far from perfect and we're thinking of ways to make gg easier to use with more peculiar build systems. If you have any suggestions or comments, please let us know!

--Sadjad

Nick S. · Answer 5 · Tue Aug 13 2019 18:52:19 GMT+0800 (China Standard Time)

@sadjad first of all, thanks for the great work and excellent explanation of our mistakes.

I have tried your instruction and everything works fine except one little thing: you should pass gg's (aka patched) g++\gcc to cmake command because CXX="g++" CC="gcc" just takes your original g++\gcc (and builds target bin without gg), so:

echo $GG_MODELPATH
# => path to gg's wrappers (see more in README)
CXX="$GG_MODELPATH/g++" CC="$GG_MODELPATH/gcc" GG_BYPASS=1 gg infer cmake <SRC-PATH>
...

And after that, everything works as it should:

We understand that this is far from perfect and we're thinking of ways to make gg easier to use with more peculiar build systems.

As far as I understand, passing CXX\CC to CMake should be a common workaround and maybe add an example to README or to examples folder? Moreover, we have a ready-made example for this above. I can take care of this if you want :)

Sadjad Fouladi · Answer 6 · Fri Aug 16 2019 04:02:29 GMT+0800 (China Standard Time)

@ssnickolay, I just created and pushed a small wrapper for CMake. Now, the user can just run gg infer cmake ../src/, and it will be handled properly.

It would still be very nice if we could have an example of using CMake in the README :)

Nick S. · Answer 7 · Fri Aug 16 2019 17:40:37 GMT+0800 (China Standard Time)

I just created and pushed a small wrapper for CMake

Ah, it is so simple, I love it!

It would still be very nice if we could have an example of using CMake in the README :)

Actually, I thought about something more reproducible. What do you think if I cover current examples via Docker? People who have not (or don't like) Docker could still run examples on local machines. But other users (who have not Ubuntu, like me) could run examples via Docker.
I thought about it because I spent a couple of hours to run the excamera example yesterday. And I still working on running viddec examples because as far as I understand part of related files contains in tools/python_sdk/examples/viddec-example folder (I mean fetch-deps.sh).
tl;dr I want to actualize examples (and add example of using CMake) and do it more comfortable to reproduce If you okay with it :)

Christoph Siedentop · Answer 8 · Sun Aug 18 2019 05:31:04 GMT+0800 (China Standard Time)

Small update: Here is a screen cast:
https://asciinema.org/a/262982 for LLVM.

@ssnickolay , I'd like to know how you got the excamera example to run?

Nick S. · Answer 9 · Sun Aug 18 2019 22:10:03 GMT+0800 (China Standard Time)

@siedentop I'll write instructions in the coming days and mention you ;)