Decentralize Ray.jl package structure
omus opened this issue · comments
Right now due to using the Bazel build system we're rather limited in what we can do in regards to setting up our various Ray.jl Julia packages. In the ideal future where we can use BinaryBuilder2 to build with Bazel our package structure would look like the following:
- ray_jll.jl: products include the ray CLI and the core worker as a shared library (patched in)
- ray_cxxwrap: contains the C++ source code for interfacing with the shared core worker library. Additionally, contains the Bazel build files and some low-level core worker wrapper code.
- ray_julia_jll.jl: Hosts the shared library created from ray_cxxwrap via artifacts. Only a separate repo from ray_cxxwrap.jl as the commits here are generated.
- Ray.jl: The Julia Ray core client side code for interacting with the core worker backend.
For development we'd make use of the ArifactOverrides.toml to let us build the source code from ray_cxxwrap.jl for use in Ray.jl. I also think we should move away from the ray_core_worker_julia_jll name as I doubt we'll have to worry about having separate JLL packages to start.
Mirroring what seems to be the pattern for CxxWrap packages. In particular Geant4.jl:
We've opted to not-decentralize the Ray.jl package as it will result in split-PRs which could slow down our development. See #124 for more details.