Is it possible to support Spark Connect?
Enochack opened this issue · comments
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As Spark Connect was introduced since Spark 3.4, it could also be a chance for query engines like Ballista. If Ballista could support the Spark Connect protocol, it will be much more smooth for Spark users to migrate their codebase. I know it's a very chanlleging task to complete. Just brainstorm.
Describe the solution you'd like
The Spark Connect client passes an unresolved logical plan to the server. So we will need to transform the logical plan into an optimized Ballista/Datafusion plan.
Describe alternatives you've considered
Additional context
Yes, it could be supported, at least to some degree. I looked at the docs, and it looks like Spark Connect is essentially a gRPC service that accepts protobuf describing Spark logical plans and then executes the plans and sends results back.
This seems achievable for anything that Spark can represent in a logical plan that Ballista also supports. I haven't read enough of the documentation to understand how UDFs would work.