Giters

rilldata / rill

Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.

Home Page:https://www.rilldata.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Upgrade to DuckDB v0.10.3 and add delta extension

AndrewRTsao opened this issue · comments

commented

DuckDB v0.10.3 adds native support for reading delta lake format:
https://duckdb.org/2024/06/10/delta.html

We should upgrade our internal duckdb to v0.10.3 and include the Delta extension in our binary so that we can read from delta lake files in object storage directly in Rill.

commented

Closing this issue because:

  1. We already have an issue for upgrading to DuckDB 1.0: #5052
  2. The delta extension is automatically downloaded and loaded by DuckDB when first used: https://duckdb.org/docs/extensions/delta. Since it's not a frequently used extension, it may not be worth bundling it directly in our binary. Feel free to reopen this issue if you think we should bundle it directly in our binary.

Note that there may be other features needed to use the DeltaLake extension with Rill for production uses, namely around setting credentials correctly for S3 (we don't currently use DuckDB's native credentials management features, but we should!).

commented

Created the following placeholder ticket so we can test (and document) this once we update duckdb: https://github.com/rilldata/FE-private-issues/issues/24