Giters

PrefectHQ / prefect-dbt

Collection of Prefect integrations for working with dbt with your Prefect flows.

Home Page:https://prefecthq.github.io/prefect-dbt/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Why this matters and technical considerations

sungchun12 opened this issue · comments

commented

Developer Pain Points:

  • I as a developer don't want to fiddle with creating custom python api calls when something more established should exist with Prefect
  • When I create my own homegrown api call from Prefect to dbt Cloud, I don't have full confidence what's a Prefect error vs. dbt Cloud error
  • I don't want to maintain integration code and/or my own api calls when the open source community may have better implementations
  • I as a developer don't know all the potential workflows dbt Cloud + Prefect can support
  • Data engineers and Analytics Engineers play baton pass and not hot potato to schedule dbt Cloud jobs elegantly via Prefect

Success Criteria:

  • "Plug and play" my dbt Cloud connection and job configuration
  • Secure: my api token isn't in plain text
  • A verified badge on the prefect registry
  • dbt Cloud jobs run successfully
  • Docs to celebrate and empower developers
    • This should also help us clarify how prefect and dbt can be complementary

Technical Considerations:

  • Store api token in environment variable?
  • Send temporary override dbt command steps that don't permanently override the job configuration: here
  • Will leverage battle-tested V2 dbt Cloud API
  • Sending useragent metadata in the API request to track Prefect usage with dbt Cloud
  • How is pagination handled in v2 dbt Cloud API?
    • you need to use the offset/limit/count and build it yourself in a loop: One main function and one that does the pagination
    • Example:GET /items?limit=20&offset=100