TobikoData / sqlmesh

Efficient data transformation and modeling framework that is backwards compatible with dbt.

Home Page:https://sqlmesh.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Full DuckDB Attach Support

eakmanrq opened this issue · comments

DuckDB allows defining not only a name to attach to but also a "type" for that name. An example is attaching a SQLite database: https://duckdb.org/docs/extensions/sqlite.html

We should consider matching dbt's format for this which would also make having this work through the adapter easier: https://github.com/duckdb/dbt-duckdb?tab=readme-ov-file#attaching-additional-databases

I think we could keep our catalogs property for this but make it be t.Dict[str, t.Union[str, t.Dict[str, str]]. This would provide a simple mapping for most cases and then a more complex option for doing the advanced attached options.

Hey @eakmanrq, I'd love to take a crack at this! (I'm @wseaton in the Tobiko slack)

@wseaton Hey Will! Sounds good! I think the trickiest part is trying to make sure we can maintain backwards compatibility since we will be changing something that is currently expected to just be a string to also support a dict too. Need to make sure Pydantic handles that properly. If not we could consider adding a new property.