z3z1ma / dbt-osmosis

Motivation

I would like to specify a dbt profile by --profile and dbt variables by --vars.

First, we have multiple profiles and select one based on the running environment. So, our default profile is set with a local development one. We would like to pass a profile to the CLI.

Second, our dbt models requires variables set by the --vars option so that we dynamically switch the Google Cloud project. We have different YAML files corresponding the destination environments respectively. We run a dbt command like dbt build --vars "$(cat vars-production.yml)". So, we would like to pass dbt variables to dbt-osmosis too.

# vars-production.yml
projects:
  project-a: "project-a-production"

# vars-staging.yml
projects:
  project-a: "project-a-staging"

This is a pseudocode to dynamically get the destination Google Cloud project ID

{% set gcp_project = var('projects')['project-a'] %}

{{
  config(
    materialized="table",
    database=gcp_project,
    schema="dwh_ai_inquiry",
    alias="answer",
  )
}}

SELECT ...

I will take care of this, thanks for the issue!

@z3z1ma Sounds good! If there is anything I can help, please let me know.

@z3z1ma can you briefly tell me how we can implement it? I took a look at the code before. I couldn't exactly understand how we modify. If I can understand that, I am interested in contributing. Thanks!

Hello 👋
@yu-iskw

The props need to be added here

dbt-osmosis/src/dbt_osmosis/vendored/dbt_core_interface/project.py

Line 205 in 26a93bb

class DbtConfiguration:

Then the appropriate kwargs need to go in the DbtProject constructor here

dbt-osmosis/src/dbt_osmosis/vendored/dbt_core_interface/project.py

Line 353 in 26a93bb

def __init__(

The DbtYamlManager inherits from the above. So from there it is straightforward. And you would just need to pass them through click.

@z3z1ma Thank you for the instruction. I will find some time to implement it.

@z3z1ma I created a pull request to support the --vars option to the sub commands of dbt-osmosis yaml. Can you review it? I will create another pull request to support the --profile option later.

#99

@z3z1ma I created the other pull request to support the --profile option in the sub commands of dbt-osmosis yaml. Can you review the pull request?

#101

Support the `--profile` and `--vars` options

Motivation