dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Home Page:https://getdbt.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Use all adapter names for adapter.dispatch

jtcohen6 opened this issue · comments

Describe the feature

The dbt-redshift adapter plugin inherits from the dbt-postgres adapter plugin. This reflects the fact that they share a lot of code: connection mechanism (psycopg2), basic syntax, types, materializations, etc. (This can be obfuscated by the fact that their shared implementations of macros are also often the default__ dbt implementations.)

Today, when dbt dispatches package_name.macro_name macro on Redshift, it looks for:

  1. my_project.redshift__macro_name
  2. my_project.redshift__macro_name
  3. package_name.postgres__concat
  4. package_name.postgres__concat

I think it should instead look for:

  1. my_project.redshift__macro_name
  2. my_project.postgres__macro_name
  3. my_project.default__macro_name
  4. package_name.redshift__macro_name
  5. package_name.postgres__macro_name
  6. package_name.default__macro_name

This is a possibility we weighed in #2679, to the point that we make it possible as a one-line change:
https://github.com/fishtown-analytics/dbt/blob/5ba5271da99bc1ef7fbad2f7d0b45634087300fa/core/dbt/context/providers.py#L104-L109

What's the risk?

There are cases today where Redshift uses the default__ implementation and Postgres uses a custom postgres__ implementation, e.g. dbt_utils.dateadd.

I don't think this is a breaking change per se, because Redshift largely supports PostgreSQL syntax, but we may want to add implementations that explicitly preserve existing behavior:

{% macro redshift__dateadd(datepart, interval, from_date_or_timestamp) %}
    {% do return(default__dateadd(datepart, interval, from_date_or_timestamp)) %}
{% endmacro %}

Who will this benefit?

This will enable maintainers of similar-yet-slightly-different plugins, e.g. dbt-sqlserver and dbt-synapse, who can then "share" macro implementations between parent and child adapters. See microsoft/dbt-synapse#18.

It's possible to imagine many more adapter inheritances in the future:

  • a dedicated dbt-databricks or dbt-delta that inherits from dbt-spark
  • a dbt-hive that inherits from dbt-spark or dbt-presto
  • a dbt-materialize that inherits from dbt-postgres

friendly, curious bump. is this still something in consideration for Margaret Mead. Relatedly, I just peeped MaterializeInc/materialize-dbt-utils. Pretty cool to see folks catching dispatch fever!

Yes, this is still something I'd like to do for Margaret Mead! The change itself is very straightforward. We'll want to add some tests for it, and it will likely require a few changes in packages, especially dbt-utils.