Upgrade to Datafusion v19
jychen7 opened this issue · comments
Background
Datafusion v19.rc1 by default turn on parallel scan on parquet files. ClickBench suggests almost 2x speed up in general.
Goal
Upgrade to Datafusion v19 .rc1
Note
can refererence to #242
cc @houqp , may need help on updating on https://github.com/delta-io/delta-rs and https://github.com/roapi/connector-x
please feel free to ping me if you need any help in updating these upstream dependencies :)