[CFP] LLVM Runtimes Build

[Petr Hosek]

Title

LLVM Runtimes Build

Author

Petr Hosek, [e-mail], Google

Abstract

While building Clang and LLVM tools is a simple and straightforward process, building a complete cross-compilation toolchain, that is, a toolchain that supports a number of different targets, can be a complicated endeavor that often involves custom scripts and infrastructure.

In particular, building LLVM runtimes like libc++ or compiler-rt can be especially challenging because runtimes have to be built with the just-built compiler for the target platform and cross-compiling these runtimes often requires custom build scripts to handle subtle bootstrap ordering issues.

In this talk, we are going to cover the LLVM runtimes build which was designed to simplify this process and allows building a complete LLVM-based cross-compiling toolchain with a single CMake invocation.

The LLVM runtimes build is already being used by a number of vendors and we hope that in the future it might become the default way of building LLVM-based toolchain distributions. If you are interested in using runtimes build, in this talk I hope to give you enough information to get you started.

[Brian Cain]

Can you share a reference build recipe or better yet a build log? I'd like to review it (details and minutia might not be covered during the talk).

[Nick Desaulniers (paternity leave)]

Thanks for taking the time to write up a CFP; we'd be overjoyed to have you present at LLVM Distributors Conf 2021! If you still plan on presenting, this is a reminder to get started on your slides for next week. Once they're done, we will contact you about submitting a PDF of your slides as either a pull request to this repository or via email to the organizer. We hope to have a schedule finalized by EOW; we may iterate on the schedule based on whether presenters have conflicts. Please keep this issue open for attendees to ask questions, or close this issue if you no longer plan on attending. Reminder to keep your talk concise (15 minutes); we wont be doing time for questions in order to fit as much content as possible. Attendees should ask questions here in this github issue.

Additionally, the committee would like you to keynote with this talk. :) WDYT?

[Petr Hosek]

I'd be happy to!

[Petr Hosek]

Can you share a reference build recipe or better yet a build log? I'd like to review it (details and minutia might not be covered during the talk).

@androm3da The CMake cache file we use to build our toolchain is in the LLVM tree. This demonstrates all the features of the runtimes build. Here is an example of a build log from one of our builders. Let me know if you have more questions.

[Martin Storsjö]

The main open question I have regarding this (I haven't dug into it so maybe it's documented somewhere?), is how to inject other outside-of-llvm build tasks, e.g. allowing installing C runtime headers after building the tools, building compiler-rt builtins, then building the libc or similar, then continuing with the rest of the runtimes. Is one supposed to do something like ninja host-tools-only vs ninja runtimes-builtins vs ninja runtimes to run things in multiple stages?

Secondly, I've seen that the runtimes build does have provisions for building the builtins separately first, before running cmake to configure the rest of the runtimes. But it does seem like one would need to split out libunwind in the same way. If building in a configuration that uses -unwindlib or CLANG_DEFAULT_UNWINDLIB set to libunwind, then all of the cmake tests for the rest of the runtimes (including libunwind) would fail. If splitting out libunwind to a separate build step, it'd be possible to inject -unwindlib=none (similar to -nostdlib++) while configuring libunwind, then separately configuring the rest of the runtimes afterwards. But the more one splits up the runtimes in stages like this, the less benefit there's from doing a single cmake run compared to individual standalone builds.

[Petr Hosek]

The main open question I have regarding this (I haven't dug into it so maybe it's documented somewhere?), is how to inject other outside-of-llvm build tasks, e.g. allowing installing C runtime headers after building the tools, building compiler-rt builtins, then building the libc or similar, then continuing with the rest of the runtimes. Is one supposed to do something like ninja host-tools-only vs ninja runtimes-builtins vs ninja runtimes to run things in multiple stages?

There isn't a way to do this currently, but it has been requested and I think we'll need to support this in the future. What I have in mind would be to provide a pair of variables, one that you could point at an outside-of-LLVM CMake file that contains your build tasks, and another one that would specify a target to depend on in that file. If set, we'll include that file from the runtimes build and update the dependency graph to make your custom target is built.

Secondly, I've seen that the runtimes build does have provisions for building the builtins separately first, before running cmake to configure the rest of the runtimes. But it does seem like one would need to split out libunwind in the same way. If building in a configuration that uses -unwindlib or CLANG_DEFAULT_UNWINDLIB set to libunwind, then all of the cmake tests for the rest of the runtimes (including libunwind) would fail. If splitting out libunwind to a separate build step, it'd be possible to inject -unwindlib=none (similar to -nostdlib++) while configuring libunwind, then separately configuring the rest of the runtimes afterwards. But the more one splits up the runtimes in stages like this, the less benefit there's from doing a single cmake run compared to individual standalone builds.

Yes, that's correct, this is one of the existing issues with the runtimes build. I believe that solution to both the builtins as well as the libunwind issues is to change the way we perform CMake checks for supported flags and libraries. The problem is that these checks by default try to compile and link a simple test program, but that's not really possible in the case of the runtimes build because we don't have a usable toolchain yet (that's what we're trying to build). For compile checks, we should switch to CMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY (another option is to use custom checks which we use for builtins). For library and link checks, we'll have to implement custom checks that would set -nostartfiles -nodefaultlibs (or the Windows equivalent). If we do this, then I believe we'll not only address the libunwind issue, but we could even build builtins together with the rest of the runtimes and rely on the CMake dependencies to enforce the correct ordering.

[Martin Storsjö]

The main open question I have regarding this (I haven't dug into it so maybe it's documented somewhere?), is how to inject other outside-of-llvm build tasks, e.g. allowing installing C runtime headers after building the tools, building compiler-rt builtins, then building the libc or similar, then continuing with the rest of the runtimes. Is one supposed to do something like ninja host-tools-only vs ninja runtimes-builtins vs ninja runtimes to run things in multiple stages?

There isn't a way to do this currently, but it has been requested and I think we'll need to support this in the future. What I have in mind would be to provide a pair of variables, one that you could point at an outside-of-LLVM CMake file that contains your build tasks, and another one that would specify a target to depend on in that file. If set, we'll include that file from the runtimes build and update the dependency graph to make your custom target is built.

Hmm - while this would work, I'm not entirely sure it's the most flexible solution for distributors in general. The GCC analogue is that you first do make all-gcc and make install-gcc, then use that to bootstrap your base environment, then return to the GCC build to do make all and make install (which builds libgcc, libstdc++). Not quite as neat, but allows you to do any amount of tasks inbetween. Much of that bootstrapping at that stage might be e.g. autotools based - I guess it's possible to add cmake wrapping of that, but it seems a bit inconvenient to have to add that.

Then again, I guess it's not that far off to do that already, with just ninja install-clang install-clang-resource-headers (and similar for e.g. llvm-ar) first and then return to do a full ninja? If there would be an umbrella target for all-minus-runtimes this would be straightforward to do.

In the case of my toolchain, the gap between building the compiler and building runtimes is not only for building some CRT base files, I also install wrappers around clang (that set all the defaults), which I use as the main compiler entry points when building the runtimes. In the runtimes build I presume there's no option for using that (e.g. invoking aarch64-w64-mingw32-clang instead of just clang -target aarch64-w64-mingw32, when the former adds defaults like -rtlib=compiler-rt) but I'd have to pile up the corresponding mandatory compiler options in variables passed down to the runtimes build? (In the standalone build today I just invoke cmake for the runtimes with CMAKE_C_COMPILER=<arch>-w64-mingw32-clang.)

Secondly, I've seen that the runtimes build does have provisions for building the builtins separately first, before running cmake to configure the rest of the runtimes. But it does seem like one would need to split out libunwind in the same way. If building in a configuration that uses -unwindlib or CLANG_DEFAULT_UNWINDLIB set to libunwind, then all of the cmake tests for the rest of the runtimes (including libunwind) would fail. If splitting out libunwind to a separate build step, it'd be possible to inject -unwindlib=none (similar to -nostdlib++) while configuring libunwind, then separately configuring the rest of the runtimes afterwards. But the more one splits up the runtimes in stages like this, the less benefit there's from doing a single cmake run compared to individual standalone builds.

Yes, that's correct, this is one of the existing issues with the runtimes build. I believe that solution to both the builtins as well as the libunwind issues is to change the way we perform CMake checks for supported flags and libraries. The problem is that these checks by default try to compile and link a simple test program, but that's not really possible in the case of the runtimes build because we don't have a usable toolchain yet (that's what we're trying to build). For compile checks, we should switch to CMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY (another option is to use custom checks which we use for builtins). For library and link checks, we'll have to implement custom checks that would set -nostartfiles -nodefaultlibs (or the Windows equivalent). If we do this, then I believe we'll not only address the libunwind issue, but we could even build builtins together with the rest of the runtimes and rely on the CMake dependencies to enforce the correct ordering.

Thanks, that sounds like good direction.

I remember playing with CMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY for libunwind/libcxx at some point, and I remember there was some gotchas with it, that kinda made it not work as desired there. But I don't remember the details and whether it'd be possible to overcome by a bit more work, or if it was a fundamental issue though.