Move Blaze/DataFusion setup to Spark executor init
yjshen opened this issue · comments
Currently, we rely on OnceCell
and lazy_static
to init tokio / DataFusion setups for the first time a task is executed. We could move these initialization steps into ExecutorPlugin.init
, and avoid passing each time we call a task execution.
pool_size: i64,
batch_size: i64,
native_memory: i64,
memory_fraction: f64,
tmp_dirs: JString,
https://spark.apache.org/docs/3.0.3/api/java/org/apache/spark/api/plugin/ExecutorPlugin.html