blaze-init / spark-blaze-extension

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Move Blaze/DataFusion setup to Spark executor init

yjshen opened this issue · comments

Currently, we rely on OnceCell and lazy_static to init tokio / DataFusion setups for the first time a task is executed. We could move these initialization steps into ExecutorPlugin.init, and avoid passing each time we call a task execution.

    pool_size: i64,
    batch_size: i64,
    native_memory: i64,
    memory_fraction: f64,
    tmp_dirs: JString,

https://spark.apache.org/docs/3.0.3/api/java/org/apache/spark/api/plugin/ExecutorPlugin.html