upgrading from previous rules
johnynek opened this issue · comments
so, I'm trying now... it isn't working as a drop in replacement.
I'm hitting a few issues:
the version that previously worked won't download:
graal_bindist_repository(
name = "graal",
java_version = "11",
version = "22.1.0",
)
changing to:
graalvm_repository(
name = "graalvm",
java_version = "11",
version = "22.1.0",
)
gives me:
INFO: Repository graalvm instantiated at:
/Users/oboykin/code/rankresearchv2/WORKSPACE:217:19: in <toplevel>
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:684:36: in graalvm_repository
Repository rule _graalvm_bindist_repository defined at:
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:547:46: in <toplevel>
ERROR: An error occurred during the fetch of repository 'graalvm':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 278, column 17, in _graal_bindist_repository_impl
fail("Unable to locate GraalVM distribution '%s' at version '%s' for platform '%s'" % (
Error in fail: Unable to locate GraalVM distribution 'oracle' at version '22.1.0' for platform 'macos-aarch64'
ERROR: /Users/oboykin/code/rankresearchv2/WORKSPACE:217:19: fetching _graalvm_bindist_repository rule //external:graalvm: Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 278, column 17, in _graal_bindist_repository_impl
fail("Unable to locate GraalVM distribution '%s' at version '%s' for platform '%s'" % (
Error in fail: Unable to locate GraalVM distribution 'oracle' at version '22.1.0' for platform 'macos-aarch64'
ERROR: /Users/oboykin/code/rankresearchv2/build_tools/bazel_rules/jar_scanner/BUILD.bazel:33:13: //build_tools/bazel_rules/jar_scanner:jarscanner-native depends on @graalvm//:native-image in repository @graalvm which failed to fetch. no such package '@graalvm//': Unable to locate GraalVM distribution 'oracle' at version '22.1.0' for platform 'macos-aarch64'
it's a bit hard for me to follow through which platforms are defined. Adding distribution = "ce" gives:
ERROR: /Users/oboykin/code/rankresearchv2/build_tools/bazel_rules/jar_scanner/BUILD.bazel:33:13: //build_tools/bazel_rules/jar_scanner:jarscanner-native depends on @graalvm//:native-image in repository @graalvm which failed to fetch. no such package '@graalvm//': Unable to locate GraalVM distribution 'ce' at version '22.1.0' for platform 'macos-aarch64'
this worked previously for us. We need to build on macos arm and linux.
The default distribution is set to "oracle" which is a change (which could open users to penalties from oracle btw. I think this is a very bad default):
rules_graalvm/internal/graalvm_bindist.bzl
Line 574 in aedd413
It seems like adding distribution = None would trigger this branch:
rules_graalvm/internal/graalvm_bindist.bzl
Line 193 in aedd413
but that fails differently:
INFO: Repository graalvm instantiated at:
/Users/oboykin/code/rankresearchv2/WORKSPACE:217:19: in <toplevel>
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:684:36: in graalvm_repository
Repository rule _graalvm_bindist_repository defined at:
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:547:46: in <toplevel>
ERROR: An error occurred during the fetch of repository 'graalvm':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 255, column 31, in _graal_bindist_repository_impl
dist_name = dist_names[ctx.attr.distribution]
Error: key "" not found in dictionary
ERROR: /Users/oboykin/code/rankresearchv2/WORKSPACE:217:19: fetching _graalvm_bindist_repository rule //external:graalvm: Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 255, column 31, in _graal_bindist_repository_impl
dist_name = dist_names[ctx.attr.distribution]
Error: key "" not found in dictionary
so I think the distribution = None may be set to the empty string since the distribution is a string? idk. It's somehow missing the branch.
I'll poke a bit more, but it looks like a lot of the code branches may not have test coverage, or at least cases we need don't.
okay...
I see the confusion now...
both version and java_version are the same. I thought one was the graalvm version and one was the java version. I don't see why we need both. And we should alert of the major version of version isn't the same as java_version.
That was very unclear to me.
with:
graalvm_repository(
name = "graalvm",
#components = [],
distribution = "ce", #"ce", # `oracle`, `ce`, or `community`
java_version = "17", # `17`, `20`, or `21`, as supported by the version provided
version = "17.0.8", # earlier version format like `22.x` also supported
)
I can download, but then I hit a new error:
INFO: Repository graalvm instantiated at:
/Users/oboykin/code/rankresearchv2/WORKSPACE:221:19: in <toplevel>
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:686:36: in graalvm_repository
Repository rule _graalvm_bindist_repository defined at:
/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl:549:46: in <toplevel>
ERROR: An error occurred during the fetch of repository 'graalvm':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 332, column 33, in _graal_bindist_repository_impl
ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/graalvm/temp12222422846218164260/graalvm-community-jdk-17.0.8_macos-aarch64_bin.tar.gz to /private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/graalvm/temp12222422846218164260: Prefix "graalvm-community-openjdk-17.0.8+9.1/Contents/Home" was given, but not found in the archive. Here are possible prefixes for this archive: "graalvm-community-openjdk-17.0.8+7.1".
ERROR: /Users/oboykin/code/rankresearchv2/WORKSPACE:221:19: fetching _graalvm_bindist_repository rule //external:graalvm: Traceback (most recent call last):
File "/private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/rules_graalvm/internal/graalvm_bindist.bzl", line 332, column 33, in _graal_bindist_repository_impl
ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/graalvm/temp12222422846218164260/graalvm-community-jdk-17.0.8_macos-aarch64_bin.tar.gz to /private/var/tmp/_bazel_oboykin/63808c00052f93fa156f2d8173529afd/external/graalvm/temp12222422846218164260: Prefix "graalvm-community-openjdk-17.0.8+9.1/Contents/Home" was given, but not found in the archive. Here are possible prefixes for this archive: "graalvm-community-openjdk-17.0.8+7.1".
making the following change I could bypass the previous issue:
--- a/internal/graalvm_bindist_map.bzl
+++ b/internal/graalvm_bindist_map.bzl
@@ -64,7 +64,7 @@ _AlignedVersions = {
_VmReleaseVersions = {
"20.0.2": "20.0.2+9.1",
"20.0.1": "20.0.1+9.1",
- "17.0.8": "17.0.8+9.1",
+ "17.0.8": "17.0.8+7.1",
"17.0.7": "17.0.7+9.1",
}
fixes I needed to get working here:
@johnynek thank you for this very thorough review. the default at oracle was not intended. I'm preparing a set of fixes based on this and your other reported issue.
btw, great point on alerting re/alignment on version and java_version
the reasoning there: GraalVM releases are not "aligned" to OpenJDK releases until the latest version. starting at the latest release, 20.x conveniently refers, always, to Java 20, and so on. obviously, this is an issue with past versions like 22.1.0 which would normally seem newer (but are in fact older).
the inner version is an interesting issue - the change for + "17.0.8": "17.0.8+7.1",. we can probably fix that by first checking a map of VM release versions specifically for the older bindist. i don't think adding to the old bindist is an issue since the new bindist is auto-generated and so we shouldn't have an ongoing maintenance burden there.
(i also must know the major java version in order to set up a java toolchain -- so perhaps we could drop that functionality for the legacy rule, which would let me drop any requirement for a java_version attribute. let me know your thoughts?)
The oracle toolchain is also broken: the mapping downloads the standard oracle jdk instead of the Graal one, and fails because the file name is completely different:
rules_graalvm/internal/graalvm_bindist_map.bzl
Lines 1529 to 1537 in aedd413
ERROR: An error occurred during the fetch of repository 'rules_graalvm~override~graalvm~graalvm':
Traceback (most recent call last):
File "/home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override/internal/graalvm_bindist.bzl", line 330, column 33, in _graal_bindist_repository_impl
ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override~graalvm~graalvm/temp10856015587010176410/jdk-17.0.8_linux-x64_bin.tar.gz to /home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override~graalvm~graalvm/temp10856015587010176410: Prefix "graalvm-jdk-17.0.8+9.1/" was given, but not found in the archive. Here are possible prefixes for this archive: "jdk-17.0.8".
ERROR: <builtin>: fetching _graalvm_bindist_repository rule //:rules_graalvm~override~graalvm~graalvm: Traceback (most recent call last):
File "/home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override/internal/graalvm_bindist.bzl", line 330, column 33, in _graal_bindist_repository_impl
ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error extracting /home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override~graalvm~graalvm/temp10856015587010176410/jdk-17.0.8_linux-x64_bin.tar.gz to /home/isaac/.cache/bazel/_bazel_isaac/995a612fe7c069966079f6b33dd349d1/external/rules_graalvm~override~graalvm~graalvm/temp10856015587010176410: Prefix "graalvm-jdk-17.0.8+9.1/" was given, but not found in the archive. Here are possible prefixes for this archive: "jdk-17.0.8".
@nuudlman hm, you're right. i don't know how that was missed. i'm sorry about that. let me fix.
@johnynek / @nuudlman this next release, 0.9.2, should fix these issues. allow me to summarize what's been reported so far below, along with fixes for each
the version that previously worked won't download [.] changing to [...] gives me [...]
I've made a few changes here:
- Alias for
graal_bindist_repository(no need to change these callsites) - Alias for
graal_binary(no need to change these callsites) - Downloads for previous GraalVM versions are fixed
- Integration test added for this exact case (it's called
legacy-rules, inexample/integration_tests)- This test will now be required for future PRs
Adding
distribution = "ce"gives [...]
The previous behavior was that passing distribution = None would actually trigger the legacy code, and this would have worked fine had I not defaulted to Oracle GVM. I guess I read their blog post about licensing and thought Oracle GVM was now the same as CE w.r.t. licensing, which is not a safe assumption. That new license is here, if curious. After reading the license itself it's not as "free" as the title might suggest, so thank you very much for catching this.
Explicitly having to pass distribution = None is awkward and was not the intended outcome. The default has been restored to None, which allows the legacy-style invocation (no value for distribution attr). If a default is needed, it now defaults to CE.
but that fails differently [...]
The previous versions should work now. I think we should keep this map because it allows eliding support for those older versioning schemes in the mapping generator (which is targeting the new release cycle). The sha mismatch seems like a bad global find/replace, so I've fixed that, of course, too.
both version and java_version are the same. I thought one was the graalvm version and one was the java version. I don't see why we need both. And we should alert of the major version of version isn't the same as java_version.
This is driven by two requirements, at least one of which we can safely drop:
-
Setting up Java toolchains. I need to know the major Java version so that I can pass it to Bazel, which uses it for the
java_runtime_versionproperty and related toolchain selection. If there is no desire to set up a Java toolchain, I can drop this logic, and document the difference. Thoughts? Is it appealing to use a GraalVM Java toolchain on previous rule versions? It would possibly save from another JDK download (depending on what's in use), but it's up to you. -
GraalVM versions which cover multiple JVMs. After the recent version alignment,
20.x.xconveniently always means "Java 20." Before this release, including the one listed here, each GraalVM engine version was also subordinated by a Java version. In these cases I think we need the Java major version in order to properly calculate the download URL, innerstrip_prefix, and so on. If I'm mistaken here, please point it out to me; these multiple version schemes are hard to coordinate in my head 😅
I can download, but then I hit a new error [...] making the following change I could bypass
It looks like GVM CE and Oracle GVM have different inner strip_prefix values, which I hadn't noticed. The prefixes are off by one character, indicating a point-release difference in the released VM version. I don't think these prefixes should change, I just think it means they may not be consistent across CE/Oracle releases. As such, I've added a second map to the bindist which covers this mapping.
The fix applied should clear up archive strip_prefix issues for both Oracle and CE.
fixes I needed to get working here
Thank you for filing -- merged -- and, I invited you @johnynek to collab on this repo. Feel free to merge an urgent fix if you need to at any time.
The oracle toolchain is also broken: the mapping downloads the standard oracle jdk
It looks like I had a copy/paste error at some point, and didn't notice because I was testing component dependency downloads on CE. Very embarrassed about this! It is now fixed, and thank you for your report as well.
Besides the fix, I've added integration tests for CE 17, CE 20, GVM 17, and GVM 20, on macOS, Linux, and Windows.
I will merge the PR attached after it passes tests, then issue a new release. Thank you both for reporting your experience!
@johnynek apologies, i missed this one
And we should alert [if the version and java_version mismatch].
Yes, I agree, to the extent this is possible. I will file an issue to cover the confusion around these properties. I can work to make java_version optional where possible and, at the very least, document why both are needed for some releases of GVM.
Thanks for your work on this.
I'll try to use main at work on Monday.
@johnynek sounds good. thank you likewise for reporting -- i deeply appreciate it.
have a great weekend