[native-image] Cross compilation support?
sureshg opened this issue · comments
Any plans to add cross-compilation feature to native-image? I know it's too early to ask for these kind of requirements since we don't even have a working Windows build. IMHO, this would be a nice feature (golang has nice support for it) to have especially if we want to build binaries for all platforms from a CI (Jenkins) machine.
Definitely would love to see this feature! This would make it so much nicer to use Java for CLI apps and servers that need to be distributed to users on Linux, Mac and Windows.
It would be amazing to have cross-compilation support. Is this still not planned for yet, or is it dependent on the Windows build? It would still be useful to have cross-compilation available to create Mac binaries on a (Linux) CI server, and then add support for Windows cross-compilation later?
@thomaswue @chrisseaton I was just wondering if this is on the roadmap for 2019, or has any ETA etc? I think it would be a game-changer!
@thomaswue @chrisseaton I was just wondering if this is on the roadmap for 2019, or has any ETA etc? I think it would be a game-changer!
Hi @thomaswue @chrisseaton since we're halfway through the year, I just thought I'd check in on this one..
Has there been any development on this issue, or any news/progress to report?
Thanks in advance! 🙂
@thomaswue Now we have java 11 and windows support ready, are there any plans to consider cross-compilation support in the near future?
Adding the support is quite tricky. The issue is that there are parts of the JDK initialized during the native image generation process that could be platform dependent and then end up in the image. One of the alternatives could be to offer a service for creating native images. Also, maybe one could use maybe a system like GitHub actions to build images for different platforms without the need to maintain/setup those machines. Would this help for your use cases or is true cross compilation a requirement?
I do think people should have mentioned this workaround elsewhere.
I am using this https://hub.docker.com/r/oracle/graalvm-ce/ docker image, to build the native-image running on Linux, on my MacBook laptop.
Would this help for your use cases
@thomaswue Thanks. Yeah, GitHub action would work for OSS projects. The issue is, our build system is mostly on Linux and wants to build native-images for windows and mac (mostly for CLI apps). So the cross-compilation would have definitely helped in this scenario.
Yeah it’s not really cross compilation if you need the target platform to build it, that’s just regular old compilation. If Cross compilation is a feature being targeted all the jvm weirdness will need to be solved, I guess the question is, is/when is it a priority?
Here is some discussion on the current limitations - https://graalvm.slack.com/archives/CN9KSFB40/p1582755160011700
Here is some discussion on the current limitations - https://graalvm.slack.com/archives/CN9KSFB40/p1582755160011700
Would it be possible to share a summary of the discussion? The slack archive can only be accessed by people that are member to the workspace.
Christian Wimmer :
No, cross compilation between different OS is more or less impossible in
the current approach because OS specific JDK classes need to be loaded
in the image generator
The biggest problem is, e.g., the file system and network stack. The JDK code
for that is very different for Linux, MacOS, and Windows. You cannot load the
code from two platforms at the same time, but you need to load the code from
the platform that the image generator runs on.
Anything on cross compiling between architectures? (x86_64 -> arm64, for example)
This is pretty huge for me. Is there any future plan at all to find a way to do this?
+1000 to Cross compilation support
+10000 to Cross compilation support
+1, it is particularly interesting for arm64 (pi) support where relying on (general) amd64 power to build would be a great enhancement.
Christian Wimmer : No, cross compilation between different OS is more or less impossible in the current approach because OS specific JDK classes need to be loaded in the image generator ....
WINE is our friend, cross compiling for windows is a big deal. Unfortunately native-image.exe and gu.exe are failing on WINE 6\ WINE7 with 0100:fixme:virtual:NtAllocateVirtualMemoryEx Ignoring 1 extended parameters 0x11fae0
upd.
Steps To reproduce:
- Build dockerfile below ``docker build -t graalvm-win .`
- Run it
docker run -it graalvm-win bash -c "wine graalvm-ce-java17-22.0.0.2/lib/installer/bin/gu.exe -h"
FROM ubuntu:21.10
RUN apt-get update \
&& apt-get install -y ca-certificates software-properties-common winbind curl unzip \
&& apt-get clean -y && rm -rf /var/lib/apt/lists/*
RUN curl --fail --silent --location --retry 3 https://dl.winehq.org/wine-builds/winehq.key | apt-key add - \
&& add-apt-repository 'deb https://dl.winehq.org/wine-builds/ubuntu/ impish main' \
&& dpkg --add-architecture i386 && apt-get update \
&& apt install -y --install-recommends xvfb winehq-devel \
&& apt-get clean -y && rm -rf /var/lib/apt/lists/*
WORKDIR /opt
ARG GRAALVM_VERSION=22.0.0.2
ARG JAVA_VERSION=java17
ARG GRAALVM_ARCH=windows-amd64
ARG GRAALVM_PKG=https://github.com/graalvm/graalvm-ce-builds/releases/download/vm-$GRAALVM_VERSION/graalvm-ce-$JAVA_VERSION-$GRAALVM_ARCH-$GRAALVM_VERSION.zip
ARG TARGETPLATFORM=linux/amd64
ENV LANG=en_US.UTF-8 \
JAVA_HOME=/opt/graalvm-ce-$JAVA_VERSION-$GRAALVM_VERSION
RUN curl --fail --silent --location --retry 3 ${GRAALVM_PKG} -o /opt/graalvm.zip && unzip -q /opt/graalvm.zip -d /opt && rm /opt/graalvm.zip
RUN wine64 wineboot --init && while pgrep wineserver > /dev/null; do sleep 1; done
CMD java -version
I was expecting the cross-compilation to be possible (when user code is 100% java) by precomputing most of the code-graph in the distro and just importing the result (so, dll, etc included) in native-image main. Can be a saner path than wine which is proven to be quite dependent of the env from my experience.
@rmannibucau it is impossible. You have to have Win build toolchain and Windows build toolchain is not crossplatform, but it works on wine (VS2019 tested). It means we can build container with all tools needed for windows build.
The only blocker is that GraalVM tools ( native-image.exe and gu.exe) don't work with WINE. I will be grateful for any help.
@bademux I understand but it means you generate sources and not .so/.dll directly whereas it is possible to generate .so/.dll directly. That said I'm fine with a cross platform toolchain while it does not rely on wine (once again I know it works on your computer, will likely work on a few others but can trivially break with any upgrade of distro so it is way too fragile for a build chain). My preference would be to rely on light windows docker image finely tagged (versions) so this is reproducible for most people and can even use native windows tools. Maybe @microsoft could help with that?
@rmannibucau it is not entirely true, you pure java code still use platform specific jvm binaries underneath.
2nd - docker is answer for what you call fragility - just freeze you dependencies in container. As well you can use easy-peasy multiplatform docker build with docker buildx, so multiple hardware platforms are covered.
3rd - ms would help here if they allow to redistribute BuildTools by changing license, so you can actually share docker image not a recipe (Dockerfile).
And last but not least and being realistic - it will be easy to solve just one blocker towards build on Windows, then just waiting for "ideal" solution.
it is not entirely true, you pure java code still use platform specific jvm binaries underneath.
yes but while the codebase on top of the jvm (understand jre excluded) does not it is fine
Agree on the other point but it should be made easier thanks to graalvm docker images IMHO, currently it is a lot of custom steps and builds and this is what we can simplify a lot IMHO.
So cross-platform seems to be rather impossible. Is there at least any hope for cross-arch compilation, like osx@amd64 -> osx@aarch64?
So cross-platform seems to be rather impossible.
Cross-compilation using native-image is not impossible. We already do it using our own fork of Graal, which allows us to cross architectures on the same OS.
Is there at least any hope for cross-arch compilation, like osx@amd64 -> osx@aarch64?
We already build native images for linux/aarch64 and linux/armv7a on a machine running linux/amd64.
@koutheir , could you please share more info on the cross-compilation? It'd help me a lot. At least in the same OS, different architecture case. Thanks a lot !
We already build native images for linux/aarch64 and linux/armv7a on a machine running linux/amd64.
It is an easy task 😜 to compile multiarch on Linux. It is doable right now without any forks, just use docker multiarch build infrastructure.
Give me true cross platform, like compiling static win binary on Linux/osX.
@mixaal, I can't really share how we do it, because it is part of a proprietary product. I simply wanted to inform you that it is possible. I'm sorry.
@bademux, we do real cross-compilation, without any containers or VMs. Doing cross-OS native-image generation is simply the next step on our way if a customer needs that.
@koutheir sounds like big work here. I'm not sure how static binary for win supposed to be cooked on Linux\osX, but finger crossed. Please share!
There is a difference between "it is possible to jump 15m height" and "it is possible to jump to the moon".
For now both options is equally impossible until someone invest $.
I can't see the difference if real and imaginary (?) crosscompilation produce the same result.
Did you investigate mingw? It is often used for C/C++ projects so even if it can require to not go directly to .so it can be an option maybe using C/C++ as intermediary language?
I'm not sure how static binary for win supposed to be cooked on Linux\osX
Binaries are just files with numbers in them. You can produce those numbers on any platform. Cross-compilation shouldn't be a weird thing.
@chrisseaton I'm not sure what to do with linking for Win static binaries on osX\linux, but anyway someone should invest time in creating tooling and support it. That is why I see more perspective in WINE + docker, then dreaming about cross platform (cross os) compilation.
@koutheir sounds like big work here.
I never said it was little work. I only said it is possible, and done.
I'm not sure how static binary for win supposed to be cooked on Linux\osX, but finger crossed.
It is possible if you follow the GNU ABI and use a mingw64 linker.
Please share!
I can't. I'm sorry.
@koutheir you source code is proprietary, but wouldn't it be possible to just share your steps for building the executable? I think a Docker-Container which just compiles a simple Hello-World app to a native Windows-Exe would be awesome and could be used as a base-image for many people, trying the same.
I understand you have a closed-source fork of GraalVM for it, but are your modifications all necessary? Maybe you could convince your employer to just contribute the parts necessary for building cross-platform images.
but wouldn't it be possible to just share your steps for building the executable?
From the point of view of a Java developer, we provide a custom native-image that produces native images for linux/armv7a on a linux/amd64. It's not a multi-step process for the Java developer.
I think a Docker-Container which just compiles a simple Hello-World app to a native Windows-Exe would be awesome and could be used as a base-image for many people, trying the same.
Again, we don't use containers or virtual machines. We modify Graal far enough to support building native images for linux/armv7a on a linux/amd64.
I understand you have a closed-source fork of GraalVM for it, but are your modifications all necessary?
Yes, they are, and they are substantial.
Maybe you could convince your employer to just contribute the parts necessary for building cross-platform images.
My employer keeps that as a commercial secret.
sorry to hear that. I know you're not using containers, but in order to share a setup with all dependencies installed, a Docker image would be a good option IMHO. However if you're unable to share your code, our only hope is other developers who come up with a solution. Thanks for your quick response.
it's 2022 still not cross compile support. really guys? with such steps hope in 2025 it will finally arrive
If Graal remains focused on servers, cross-compilation might never be implemented, because the assumption will be that the target computer will always be good enough to compile Graal itself and generate native images that it will run. If Graal starts targeting embedded systems, then cross-compilation will be one of the first things to implement.
@koutheir not sure this reasoning works because it assumes you deliver a native image of an application you own - path to do it for a vendor is way too hard as of today, in particular to maintain and validate. So if not possible it means CLI/embedded will get a harder time than needed but also that native will also be harder for products (external apps, including oss). So hope it gets enabled for both cases.
Our GitHub Action for GraalVM supports creating native images for different platforms simultaneously. Here is an example configuration to create an image each for MacOS, Windows, and Linux: https://github.com/marketplace/actions/github-action-for-graalvm#building-a-helloworld-with-graalvm-native-image-on-different-platforms
There can be differences between operating systems how the JDK or third-party library dependencies are initialized and those end up snapshotted into the native image. Therefore, the problem of cross compilation for native image is more difficult to solve than cross compilation for e.g. C programs.
We plan to provide cross-compilation support for different architectures - i.e., amd64/aarch64. There are currently no plans however to provide such cross-compilation support for different operating systems - i.e., creating a Windows binary from Linux or vice versa. In general our strategy to simplify creation of native images goes more into the direction of providing native image generation services where one can specify the desired architecture.
We plan to provide cross-compilation support for different architectures - i.e., amd64/aarch64. There are currently no plans however to provide such cross-compilation support for different operating systems - i.e., creating a Windows binary from Linux or vice versa. In general our strategy to simplify creation of native images goes more into the direction of providing native image generation services where one can specify the desired architecture.
@thomaswue Do we have any timelines or ETA for providing cross compilation for different architectures?
What is the primary source/destination architecture you would be interested in to cross-compile, is it Linux/amd64 to Linux/aarch64? Is it possible to solve your use case with the GitHub Action workaround?
I would like to build installers (or, worst case, single binaries) for windows/amd64, linux/amd64, darwin/amd64, darwin/arm64 all from a single platform, preferably linux/amd64
For windows, historically i've used launch4j + nsis
For linux, a tar file is more than sufficient (since you can trivally start a jar from a shell script)
For darwin/MacOS jpackage sort of works but not really
@thomaswue
I'm interested in Win64, linux cross arch compilation is easy achievable with docker cross build (buildx)
I've been involved in multiple architecture porting projects and the biggest use-case is cross-building Linux amd64 -> arm64 and the other way around (arm64 -> amd64) where the first one is most used since there are biggest/fastest amd64 boxes.
If possible, cross-OS building would be desireable like linux/amd64 -> darwin/amd64 as Golang currently does (it actually does any cross-building like linux/amd64 -> darwin/arm64 and so on).
Still hoping for sparc.
Hi,
from Linux/amd64 to windows|arm64 for me.
Any CI is not an option on some env so would be great to have a solution close to c++ solutions.
What is the primary source/destination architecture you would be interested in to cross-compile, is it Linux/amd64 to Linux/aarch64? Is it possible to solve your use case with the GitHub Action workaround?
I think the real advantage of cross compilation comes when JVM languages/frameworks adopt GraalVM Ex - Quarkus. Lets assume I wanna deploy an azure function using my framework(which is a perfect usecase where AOT shines). As far as I know, Azure functions only supports executable with linux/amd64. The language/framework can wrap graalvm and generate that executable for the user by hiding that complexity from user.
So the github actions workaround would not work in this case as it needs to happen from the framework's side, not the users side. The closest workaround i have is building it using docker using buildx. The issue with this is the docker dependency. Still, even with docker, its hard to do the cross compilation for different operating systems.
For enterprise deployments, it may suffice in many contexts for a vendor to utilize a server for building each new release of a project. However, cross building is extremely standard for deployments of open-source platforms and packages. It is hard to see how a tool would gain much traction for open source projects these days if it constrains the target and build environment to be the same platform or machine family.
@brainchild0 Agreed. We believe however that many open source (and even commercial) projects can use our GitHub Action for GraalVM to build their releases. Is this not a reasonable workaround? https://github.com/marketplace/actions/github-action-for-graalvm
We could also maybe build an Oracle cloud service for building.
Is this not a reasonable workaround?
@thomaswue The question has no general answer. In some cases it may be, others not. My remark directed at cases for which the answer would be negative, while acknowledging others for which it would be more optimistic.
For some open source components or platforms, you build everything in one place from scratch (or take binaries if available, but reserve the capability to build). That's just how it is.
What is worth considering is that the limitation probably will make adoption more reluctant even in border cases, due to concerns about dependence on a tool limited in broader adoptability.
We could also maybe build an Oracle cloud service for building.
Such a process is suited to projects supported by the community but still distributed as prebuilt binaries targeting major platforms. It is obviously a common case, but not the only.
Not having any useful background on the subject, I am curious to ask about feasibility for running the tool version corresponding to the target platform inside Qemu.
We believe however that many open source (and even commercial) projects can use our GitHub Action for GraalVM to build their releases. Is this not a reasonable workaround?
No because most of this solution wouldn't be free or too expensive, plus not usable in dev - without referencing projects not using Github, including on premise projects using Gitea/Drone or alike solutions. The key point is not NOT depend on a CI to be able to build. There are cases you want the release to be signed by an employe/user and not a technical key so for these cases you need an offline solution and hopefully local only (ie not multiple VM).
There can be differences between operating systems how the JDK or third-party library dependencies are initialized and those end up snapshotted into the native image. Therefore, the problem of cross compilation for native image is more difficult to solve than cross compilation for e.g. C programs.
@thomaswue So the NI tool interrogates the memory state of the running JDK, and captures a representation for inclusion into the build target? Is the captured image specific to the application being translated, or only to the combination of kernel and machine types?
What is the primary source/destination architecture you would be interested in to cross-compile, is it Linux/amd64 to Linux/aarch64? Is it possible to solve your use case with the GitHub Action workaround?
For me it would be compiling macos/aarch64 on macos/x86_64
Because GitHub actions does not have aarch64 macos runners, and for an OSS hobby project I do not want to pay for hosted Apple Silicon runners
Since all new macs are aarch64 it no longer seems worth building native images with Graal as I had been, since I can only support obsolete platforms that way
There are cases you want the release to be signed by an employe/user and not a technical key so for these cases you need an offline solution and hopefully local only (ie not multiple VM).
It's worth separating out the build from the release here. Whilst native-image can't cross-build, Conveyor can cross-package and cross-sign and is free for open source projects. So your pipeline could look like this:
- Develop on regular JVM.
- Compile with native-image using a multi-platform CI system (yes I know this is painful - I built one with home hardware to do this).
- Write a Conveyor config that pulls the artifacts from your CI outputs and then produces signed, notarized binaries and packages from whatever local hardware you have your keys on. It supports HSMs.
@mikehearn main point is to request a solution not depending on any CI like solution which are workarounds to this issue and do not cover all build/releases cases (not merging both cases but listing both as distinct but valid).
Yes, but that may not arrive any time soon and wouldn't solve the signing or packaging aspects, you'd still need other tools for that. I mention it here as an answer to your second part, so people who want to distribute CLI or GUI tools to desktops with local signing can know where to look.
@brainchild0 The state is specific to the application being translated.
In addition to this, we are depending on the local C developer tool chain a lot, so whatever cross-compilation combination we should support also needs to be available from that tool chain.
There is also the option to build within a docker container.
So, when looking at the existing workarounds such as Docker, GitHub Actions, Conveyor and the complexity of supporting all source/target platform combinations (with 5 supported platforms that would be 25), it seems like it would not be the best prioritization of engineering effort for the project at the moment. We will look more into build server setups that we can make available. Such a scenario can have additional advantages such as faster builds.
All the mentionned solutions are at least as bad as requiring a local stack and most are more impacting like requiring to oss the project, to pay or just to not be able to run locally. All that are blockers to a solution to this issue so please invest in cross chains more than platforms which are just the current workaround.
In addition to this, we are depending on the local C developer tool chain a lot, so whatever cross-compilation combination we should support also needs to be available from that tool chain.
Such a constraint by itself is not particularly limiting. Cross-compilation solutions have evolved quite substantially in the past decade, and the dissociation between build environment and build target is now rather common, and available without penalty.
it seems like it would not be the best prioritization of engineering effort for the project at the moment.
As previously stated, it is a matter of context and perspective . For some projects, remote builds would be a tremendous help. For others, it has no value. I think it is not completely accurate that CI is a reasonable substitute for local availability of cross-building operations, in the general case.
.
I wonder, is there any support for cross-compilation?