ray-project / kuberay

A toolkit to run Ray applications on Kubernetes

KubeRay

KubeRay is a powerful, open-source Kubernetes operator that simplifies the deployment and management of Ray applications on Kubernetes. It offers several key components:

KubeRay core: This is the official, fully-maintained component of KubeRay that provides three custom resource definitions, RayCluster, RayJob, and RayService. These resources are designed to help you run a wide range of workloads with ease.

• RayCluster: KubeRay fully manages the lifecycle of RayCluster, including cluster creation/deletion, autoscaling, and ensuring fault tolerance.

• RayJob: With RayJob, KubeRay automatically creates a RayCluster and submits a job when the cluster is ready. You can also configure RayJob to automatically delete the RayCluster once the job finishes.

• RayService: RayService is made up of two parts: a RayCluster and a Ray Serve deployment graph. RayService offers zero-downtime upgrades for RayCluster and high availability.

Community-managed components (optional): Some components are maintained by the KubeRay community.

• KubeRay APIServer: It provides a layer of simplified configuration for KubeRay resources. The KubeRay API server is used internally by some organizations to back user interfaces for KubeRay resource management.

• KubeRay Python client: This Python client library provides APIs to handle RayCluster from your Python application.

• KubeRay CLI: KubeRay CLI provides the ability to manage KubeRay resources through command-line interface.

Documentation

From September 2023, all user-facing KubeRay documentation will be hosted on the Ray documentation. The KubeRay repository only contains documentation related to the development and maintenance of KubeRay.

Quick Start

• RayCluster Quickstart
• RayJob Quickstart
• RayService Quickstart

Examples

• Ray Train XGBoostTrainer on Kubernetes (CPU-only)
• Train PyTorch ResNet model with GPUs on Kubernetes
• Serve a MobileNet image classifier on Kubernetes (CPU-only)
• Serve a StableDiffusion text-to-image model on Kubernetes
• Serve a text summarizer on Kubernetes
• RayJob Batch Inference Example

Kubernetes Ecosystem

• Ingress: AWS Application Load Balancer, GKE Ingress, Nginx
• Using Prometheus and Grafana
• Profiling with py-spy
• KubeRay integration with Volcano
• Kubeflow: an interactive development solution
• MCAD: A Kubernetes Solution for Queuing and Gang Dispatching jobs on Single or Multi-Cluster environment

External Blog Posts

1. Evolving Niantic AR Mapping Infrastructures with Ray Niantic (September 6, 2023)
2. Building a Modern Machine Learning Platform with Ray at Samsara Samsara (August 29, 2023)
3. Using Ray on Kubernetes with KubeRay at Google Cloud Google (August 15, 2023)
4. How DoorDash Built an Ensemble Learning Model for Time Series Forecasting with KubeRay Doordash (June 20, 2023)
5. AI/ML Models Batch Training at Scale with Open Data Hub Red Hat (May 15, 2023)
6. A cloud-native, open-source stack for accelerating foundation model innovation IBM (May 9, 2023)
7. Distributed Machine Learning at Instacart Instacart (March 17, 2023)
8. Unleashing ML Innovation at Spotify with Ray Spotify (February 1, 2023)
9. Best Practices For Ray Cluster On ACK Alibaba Cloud (Mar 12, 2024)

Talks

1. Supercharge Your AI Platform with KubeRay Anyscale + Google (November 8, 2023)
2. Sailing Ray Workloads with KubeRay and Kueue in Kubernetes Volcano + DaoCloud (October 17, 2023)
3. Serving Large Language Models with KubeRay on TPUs Google (September 19, 2023)
4. KubeRay: A Ray Cluster Management Solution on Kubernetes Anyscale (September 18, 2023)
5. The Different Shades of using KubeRay with Kubernetes Microsoft (September 18, 2023)
6. On-Demand Ray Clusters in ML Workflows via KubeRay & Sematic Sematic (September 18, 2023)
7. KubeRay - A Kubernetes Ray Clustering Solution Microsoft (February 8, 2023)
8. KubeRay x Flyte Integration Flyte (August 24, 2022)
9. Operationalizing Ray Serve on Kubernetes Anyscale (August 24, 2022)

Helm Charts

KubeRay Helm charts are hosted on the ray-project/kuberay-helm repository. Please read kuberay-operator to deploy the operator and ray-cluster to deploy a configurable Ray cluster. To deploy the optional KubeRay API Server, see kuberay-apiserver.

# Add the Helm repo
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update

# Confirm the repo exists
helm search repo kuberay --devel

# Install both CRDs and KubeRay operator v1.1.0.
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0

# Check the KubeRay operator Pod in `default` namespace
kubectl get pods
# NAME                                READY   STATUS    RESTARTS   AGE
# kuberay-operator-6fcbb94f64-mbfnr   1/1     Running   0          17s

Development

Please read our CONTRIBUTING guide before making a pull request. Refer to our DEVELOPMENT to build and run tests locally.

Getting Involved

Join Ray's Slack workspace, and search the following public channels:

• #kuberay-questions (KubeRay users): This channel aims to help KubeRay users with their questions. The messages will be closely monitored by the Ray and KubeRay maintainers.

• #kuberay-discuss (KubeRay contributors): This channel is for contributors to discuss what to do next with KubeRay (e.g. issues, pull requests, feature requests, design docs, KubeRay ecosystem integrations). All KubeRay maintainers and core contributors are in the channel.

Security

If you discover a potential security issue in this project, or think you may have discovered a security issue, we ask that you notify KubeRay Security via our Slack Channel. Please do not create a public GitHub issue.

License

This project is licensed under the Apache-2.0 License.