Leverage existing integration work from the community

Question

Leverage existing integration work from the community

sungchun12 opened this issue 2 years ago · comments

Sung Won Chung commented 2 years ago

Leverage existing work based on git repo: here

Sung Won Chung · Answer 1 · Fri May 27 2022 22:37:15 GMT+0800 (China Standard Time)

I haven't taken a thorough look, but glad to know someone's already thinking about this!

Sung Won Chung · Answer 2 · Fri May 27 2022 22:38:54 GMT+0800 (China Standard Time)

@AlessandroLollo can you provide context on your approach and what you think is best to be reused for this official integration vs. not?

ale · Answer 3 · Wed Jun 01 2022 19:13:15 GMT+0800 (China Standard Time)

Hey @sungchun12 👋
The original integration between Prefect and dbt Cloud was introduced in Prefect 1.0.
Here you can find the task.

Then, I decided to implement the same feature for Prefect 2.0.
At its core, the task is basically a wrapper around the run job API available in dbt Cloud.
The task offers some convenient features, i.e: specifying a custom domain, or waiting for the job run to complete.

The run_job task is pretty stable and I think it can be used as-is.

I've also started working on a new task create_job which should make it even easier to create a new dbt Cloud job.
I did not fully tested it in a real use case though.

Sung Won Chung · Answer 4 · Wed Jun 01 2022 23:33:00 GMT+0800 (China Standard Time)

Hey!

Thanks a bunch for adding context here. @desertaxle I recommend reading through the above and porting over code where appropriate. Hopefully, this will save everyone time so we don't have reinvent the wheel where we don't have to.