Connecting Elyra with Ray
DhavalRepo18 opened this issue · comments
As a Data scientist, I would like to use Elyra with Ray (now also part of ODH) as the backend. Is there any plan to support Ray as a backend for the execution of the pipeline? I am happy to walk over the additional req.
Hi @DhavalRepo18 - there are no immediate plans to support Ray, but I'd like to gather some more information about your use case.
Is the idea that you'd essentially be running in "local mode" (not KFP or Airflow) but that the cluster in which each notebook-based node runs is a Ray cluster? Do you need support for Python or R nodes and that they too run on Ray?
If the answer to the first question is "yes", and the second question is "no" then one way this may be achieved is to write a process proxy for Enterprise Gateway (or a gateway provisioner) that manages a kernel's lifecycle within a Ray cluster.