Credentials missing when using SnowflakeTargetConfigs in CLIProfile

[Timo S.]

Expectation / Proposal

I define four block entities (Snowflake credentials, Snowflake connector, Snowflake target config and dbt CLI profile) and link them together. In this case credentials can be changed via block configuration.

I expect that the CLI profile holds the credential/connection information as well but it didn't. This is maybe related to the TargetConfigs which can't be linked to another block entity.

Traceback / Example

from prefect_snowflake.credentials import SnowflakeCredentials
from prefect_snowflake.database import SnowflakeConnector

from prefect_dbt.cli.credentials import DbtCliProfile
from prefect_dbt.cli.configs import SnowflakeTargetConfigs


if __name__ == "__main__":
    credentials = SnowflakeCredentials(
        user="user",
        password="password",
        account="account.region.aws",
        role="role",
    )
    credentials.save("sf-cred", overwrite=True)

    connector = SnowflakeConnector(
        schema="public",
        database="database",
        warehouse="warehouse",
        credentials=SnowflakeCredentials.load("sf-cred"),
    )
    connector.save("sf-conn", overwrite=True)

    target_configs = SnowflakeTargetConfigs(
        connector=SnowflakeConnector.load("sf-conn"), schema="TEST"
    )
    target_configs.save("dbt-targetconf", overwrite=True)

    dbt_cli_profile = DbtCliProfile(
        name="jaffle_shop",
        target="dev",
        target_configs=SnowflakeTargetConfigs.load("dbt-targetconf"),
    )
    dbt_cli_profile.save("example", overwrite=True)

    dbt_cli_profile = DbtCliProfile.load("example").get_profile()

    print(dbt_cli_profile)

Output:
{'config': {}, 'jaffle_shop': {'target': 'dev', 'outputs': {'dev': {'type': 'snowflake', 'schema': 'TEST', 'threads': 4}}}}

• I would like to help contribute a pull request to resolve this!

[Andrew]

Thanks for the details! I believe this has been fixed in #115.

[Timo S.]

So we need to wait for a new release then.

[Timo S.]

Hi @ahuang11 ,
I've tested the behavior with the latest main branch and this doesn't work for me when we specifiy a schema in the SnowflakeConnector (which is required) and in the Snowflake Target Config (which is optional).
This ends up in a ValueError: The keyword, schema, has already been provided in TargetConfigs; remove duplicated keywords to continue error.

As a workaround we could only specify it in Snowflake Connector but if we use the same Connector for different dbt Configs (or may for other things) it would be a better solution to overwrite the schema from Snowflake Connector with the schema from the target config.

Furthermore it seems that the private_key_path is not written in the profiles file as expected:

config: {}
iris:
  outputs:
    dev:
      account: account.region.azure
      authenticator: snowflake
      database: dbname
      private_key_path: !!python/object/apply:pathlib.PosixPath
      - /
      - workspace
      - tmp
      - rsa_key.p8
      role: development
      schema: abc
      threads: 8
      type: snowflake
      user: username
      warehouse: wh
  target: dev

which ends in an runtime error

Encountered an error:
Runtime Error
  
  dbt encountered an error while trying to read your profiles.yml file.
  
  Runtime Error
    Syntax error near line 8
    ------------------------------
    5  |       account: account.region.azure
    6  |       authenticator: snowflake
    7  |       database: dbname
    8  |       private_key_path: !!python/object/apply:pathlib.PosixPath
    9  |       - /
    10 |       - workspace
    11 |       - tmp
    
    Raw Error:
    ------------------------------
    could not determine a constructor for the tag 'tag:yaml.org,2002:python/object/apply:pathlib.PosixPath'
      in "<unicode string>", line 8, column 25

[Andrew]

Thank you for reporting this. I'll try to get this fixed soon.