Prefect dbt 0.2.6 coroutine error

Question

Prefect dbt 0.2.6 coroutine error

lucienfregosibodyguard opened this issue 2 years ago · comments

Following this issue Sadly it's still doesn't work, got a new error

Encountered exception during execution:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1339, in orchestrate_task_run
    result = await task.fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect_dbt/cli/commands.py", line 139, in trigger_dbt_cli_command
    yaml.dump(profile, f, default_flow_style=False)
  File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/usr/local/lib/python3.10/site-packages/yaml/representer.py", line 317, in represent_object
    reduce = data.__reduce_ex__(2)
TypeError: cannot pickle 'coroutine' object

I guess it's related to the async function definition but don't know how to fix it

Andrew commented a year ago

Done!

Andrew · Answer 1 · Fri Dec 09 2022 00:09:58 GMT+0800 (China Standard Time)

Do you have this wrapped in a flow? Also are you working in notebook?

Here's something you can try in notebook / script.

from prefect import flow
from prefect_gcp import GcpCredentials
from prefect_dbt.cli import DbtCliProfile
from prefect_dbt.cli.configs import BigQueryTargetConfigs


@flow
def test_flow():
    credentials = GcpCredentials()

    target_configs = BigQueryTargetConfigs(
        credentials=credentials,
        schema="schema",
        project="project",
    )
    dbt_cli_profile = DbtCliProfile(
        name="jaffle_shop",
        target="dev",
        target_configs=target_configs,
    )
    profile = dbt_cli_profile.get_profile()
    return profile

test_flow()

Lucien Fregosi · Answer 2 · Fri Dec 09 2022 16:55:09 GMT+0800 (China Standard Time)

No the code is directly run within a kubernetes pod.

Locally it seems to work

Andrew · Answer 3 · Sat Dec 10 2022 01:30:17 GMT+0800 (China Standard Time)

Can you try to see if this branch potentially fixes the issue? I made the base class method also sync compatible, but unsure if it'll help.

https://github.com/PrefectHQ/prefect-dbt/pull/105/files

Lucien Fregosi · Answer 4 · Mon Dec 12 2022 21:58:21 GMT+0800 (China Standard Time)

I have the same issue sadly :/

Finished in state Failed(message="Flow run encountered an exception. TypeError: cannot pickle 'coroutine' object\n", type=FAILED, result=TypeError("cannot pickle 'coroutine' object"))

Andrew · Answer 5 · Tue Dec 13 2022 07:25:56 GMT+0800 (China Standard Time)

I suspect it's due to PrefectHQ/prefect-gcp#80 not yet released. Could you try pulling prefect-gcp main?

e.g. in requirements.txt prefect-gcp @ git+https://github.com/PrefectHQ/prefect-gcp.git@main

Lucien Fregosi · Answer 6 · Tue Dec 13 2022 18:49:23 GMT+0800 (China Standard Time)

Same issue with the prefect-gcp version 0.1.8+2.g28f72ba

Andrew · Answer 7 · Wed Dec 14 2022 02:47:41 GMT+0800 (China Standard Time)

To clarify: "No the code is directly run within a kubernetes pod."

Is your code wrapped in a flow?

Lucien Fregosi · Answer 8 · Wed Dec 14 2022 16:40:27 GMT+0800 (China Standard Time)

Sure all the code is wrapped in a flow and triggered with prefect

Andrew · Answer 9 · Thu Dec 15 2022 05:13:15 GMT+0800 (China Standard Time)

Thanks for confirming and I appreciate your help testing! I made another commit on https://github.com/PrefectHQ/prefect-dbt/pull/105/files

Can you help me test that again? If this doesn't work, I think we might have to make it sync again.

Lucien Fregosi · Answer 10 · Tue Dec 20 2022 16:13:18 GMT+0800 (China Standard Time)

@ahuang11 I'm so sorry It still doesn't work :( with same error

Andrew · Answer 11 · Wed Dec 21 2022 04:44:12 GMT+0800 (China Standard Time)

Thanks for testing. Can you provide a minimal reproducible example that we can try on our end?

Lucien Fregosi · Answer 12 · Tue Dec 27 2022 22:05:15 GMT+0800 (China Standard Time)

hey @ahuang11
Actually I got this error even with the previous authentication method.

The 0.2.6 is broken for us because of the co routine error

It can be hard to reproduce the error because it happens only in a Kubernetes context (it works well locally)

Package a docker image based on prefect 2.7.0-python3.10 with this version
prefect-dbt = "0.2.6"
dbt-bigquery = "1.3.0"
prefect-gcp = "0.1.6"
Add code to the docker image
the code can be taken from the official prefect gcp example the error raises with any prefect_dbt.cli.commands import trigger_dbt_cli_command
Create a storage & kubernetes block based on the previous docker image which will be used by the flow
Start a flow in the prefect UI

If it's too hard for you to replicate this, we can plan a call or even let you use my mac remotely

Andrew · Answer 13 · Sat Dec 31 2022 05:45:36 GMT+0800 (China Standard Time)

Thanks for that; can you try #111 to see if this fixes the issue? I dropped the async.

Lucien Fregosi · Answer 14 · Mon Jan 02 2023 18:23:46 GMT+0800 (China Standard Time)

@ahuang11 yes now it works perfectly with the last fix

Lucien Fregosi · Answer 15 · Tue Jan 03 2023 17:41:36 GMT+0800 (China Standard Time)

It it possible to merge and release a 0.2.8 package ? :)

Andrew · Answer 16 · Wed Jan 04 2023 01:35:09 GMT+0800 (China Standard Time)

Yes I'll try to get the PR ready soon!

Andrew · Answer 17 · Thu Jan 12 2023 02:39:51 GMT+0800 (China Standard Time)

@lucienfregosibodyguard by any chance, are you loading a DbtCliProfile like shown in #114?

Lucien Fregosi · Answer 18 · Tue Jan 17 2023 00:05:10 GMT+0800 (China Standard Time)

Here is my DbtCliProfile Definition

    def _get_credentials(self) -> GcpCredentials:
        return GcpCredentials()

    def _get_target_configs(self) -> BigQueryTargetConfigs:
        return BigQueryTargetConfigs(
            schema=self.dataset_name, project=self.gcp_project, credentials=self._get_credentials()
        )

    def _get_dbt_cli_profile(self) -> DbtCliProfile:
        return DbtCliProfile(
            name=self.dbt_folder_name, target=self.environment, target_configs=self._get_target_configs()
        )

let me know if it's not the up to date version to load DbtCliProfile

Andrew · Answer 19 · Wed Jan 18 2023 01:07:37 GMT+0800 (China Standard Time)

Okay so you don't use .save or .load?

Lucien Fregosi · Answer 20 · Wed Jan 18 2023 21:38:00 GMT+0800 (China Standard Time)

No I can try with save and load

Lucien Fregosi · Answer 21 · Wed Jan 18 2023 22:19:14 GMT+0800 (China Standard Time)

It seems better actually i don't have the coroutine error, it starts the DBT command and I got this

�[0m13:50:46  Encountered an error while reading profiles:
�[0m13:50:46    ERROR: Runtime Error
  Credentials in profile "dbt_models", target "" invalid: 'method' is a required property

Which is a "normal" error that should be solved in your PR

Lucien Fregosi · Answer 22 · Mon Feb 06 2023 18:32:50 GMT+0800 (China Standard Time)

Hi @ahuang11 could you release a new tag to easily test this and update my code ?

Andrew · Answer 23 · Tue Feb 07 2023 09:32:30 GMT+0800 (China Standard Time)

Actually this isn't fixed yet. Will try to get a new release soon...

Lucien Fregosi · Answer 24 · Thu Feb 16 2023 17:26:15 GMT+0800 (China Standard Time)

@ahuang11 could we make a new tag with the last fix ?

Lucien Fregosi · Answer 25 · Fri Feb 17 2023 17:33:48 GMT+0800 (China Standard Time)

Thaaaaaaaks it works like a charm awesome