Policy on large prebuilt libraries on crates.io?

[Mads Marquart]

Greetings! @silvanshade and I are discussing prebuilding Clang/LLVM libraries in madsmtm/objc2#345, and I have a few questions about that for the crates.io team.

Motivation

There exists several tools in the ecosystem that uses libclang, usually to do some sort of C/C++ to Rust transformation. Prominent examples include bindgen, autocxx and my own tool for generating icrate.

This is nice because libclang can be installed fairly easily on most operating systems, and the API is quite stable, even between major versions. It is, sadly, also limited in the kind of semantic analysis you can do, since there are many C/C++ quirks and attributes that are not exposed in its API.

So instead of using libclang, another option would be to use Clang/LLVM libraries directly. This unfortunately forces downstream users of your crate to somehow get/build a compatible version of Clang/LLVM libraries outside of Cargo, which is difficult and annoying for users, and has issues with reproducibility. This is the approach taken by c2rust and rustc itself (although the bootstrap script mitigates it by providing prebuilt libraries (example url)).

Proposed solution

To make it easier to use Clang/LLVM libraries directly from Rust, I propose developing a crate that helps ensure that the desired Clang/LLVM libraries is available (potentially integrated with the existing llvm-sys crate). This crate will bundle the required sources in the crate itself (a quick test says this will be around ~30MB).

These sadly take a long time to compile, so to mitigate that issue, the crate will supply prebuilt libraries for common target platforms (of course, user configurable, I'm aware of the serde_derive suddenly shipping a binary debacle a while ago).

These will also be uploaded to crates.io, and the correct libraries will be chosen by Cargo in a similar manner as is done for windows-sys. The size of the LLVM libraries are ~150MB, Clang libraries ~85MB and maybe also Clang binaries ~30MB (all after being compressed by Cargo). We would need one crate per supported target, so probably a few GB in total. @silvanshade may be able to answer if these can be mader smaller.

The current upload size limit is 10MiB, which the proposed crates would clearly hit.

Questions

What is your policy on prebuilt libraries on crates.io?

What is your policy on increasing this size limit for a crate?

What about for something like this, which is important for bindgen (an official Rust project), and has the potential to vastly improving Rust's user story on interacting with C/C++/Objective-C?

I can see that storing several GBs of data in total is troublesome, but is there another number you would be more comfortable with? (If it helps, we would be limiting ourselves (barring critical bugs) to releasing only major versions of Clang/LLVM, to reduce the strain on crates.io's servers).

If this is not acceptable, which alternative would the crates.io team recommend that we do instead? (Either from the list below, or other alternatives).

Alternatives

1. We could host these ourselves on GitHub (already done in here).
   • Requires downloading these in a build script, which is bad for compile times, offline users and sandboxing.
2. Every tool that needs Clang/LLVM libraries could provide their own prebuilt dylib (which could reduce the required binary size because of dead code elimination).
   • Possible, depends on what your
3. Only allow building from source.
   • Painfully slow, though workable.
4. Include Clang/LLVM libraries as a rustup component.
   • Difficult to ensure that the correct version is installed. Updating Rust now might break crates.
5. Include bindgen as a rustup component.
   • Requires a stable bindgen v1.0.

Prior art

Previously discussed in rust-lang/rust-bindgen#918.

The openssl crate has an option to build OpenSLL from sources, which are distributed via. openssl-src. The download size for that is ~8MB.